David S. Miller f36391d279 sparc64: Fix race in TLB batch processing. ... As reported by Dave Kleikamp, when we emit cross calls to do batched TLB flush processing we have a race because we do not synchronize on the sibling cpus completing the cross call. So meanwhile the TLB batch can be reset (tb->tlb_nr set to zero, etc.) and either flushes are missed or flushes will flush the wrong addresses. Fix this by using generic infrastructure to synchonize on the completion of the cross call. This first required getting the flush_tlb_pending() call out from switch_to() which operates with locks held and interrupts disabled. The problem is that smp_call_function_many() cannot be invoked with IRQs disabled and this is explicitly checked for with WARN_ON_ONCE(). We get the batch processing outside of locked IRQ disabled sections by using some ideas from the powerpc port. Namely, we only batch inside of arch_{enter,leave}_lazy_mmu_mode() calls. If we're not in such a region, we flush TLBs synchronously. 1) Get rid of xcall_flush_tlb_pending and per-cpu type implementations. 2) Do TLB batch cross calls instead via: smp_call_function_many() tlb_pending_func() __flush_tlb_pending() 3) Batch only in lazy mmu sequences: a) Add 'active' member to struct tlb_batch b) Define __HAVE_ARCH_ENTER_LAZY_MMU_MODE c) Set 'active' in arch_enter_lazy_mmu_mode() d) Run batch and clear 'active' in arch_leave_lazy_mmu_mode() e) Check 'active' in tlb_batch_add_one() and do a synchronous flush if it's clear. 4) Add infrastructure for synchronous TLB page flushes. a) Implement __flush_tlb_page and per-cpu variants, patch as needed. b) Likewise for xcall_flush_tlb_page. c) Implement smp_flush_tlb_page() to invoke the cross-call. d) Wire up global_flush_tlb_page() to the right routine based upon CONFIG_SMP 5) It turns out that singleton batches are very common, 2 out of every 3 batch flushes have only a single entry in them. The batch flush waiting is very expensive, both because of the poll on sibling cpu completeion, as well as because passing the tlb batch pointer to the sibling cpus invokes a shared memory dereference. Therefore, in flush_tlb_pending(), if there is only one entry in the batch perform a completely asynchronous global_flush_tlb_page() instead. Reported-by: Dave Kleikamp <dave.kleikamp@oracle.com> Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Dave Kleikamp <dave.kleikamp@oracle.com>		2013-04-19 17:26:26 -04:00
..
Kbuild	sparc: use asm-generic version of types.h	2013-03-31 19:29:13 -04:00
agp.h	…
apb.h	…
asm-offsets.h	…
asm.h	…
asmmacro.h	sparc32: add support for run-time patching of leon/sun single instructions	2012-05-27 23:52:43 -07:00
atomic.h	…
atomic_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
atomic_64.h	sparc: Support atomic64_dec_if_positive properly.	2012-11-09 19:37:59 -08:00
auxio.h	…
auxio_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
auxio_64.h	…
backoff.h	sparc64: Improvde documentation and readability of atomic backoff code.	2012-10-28 13:04:47 -07:00
barrier.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
barrier_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
barrier_64.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
bbc.h	…
bitext.h	…
bitops.h	…
bitops_32.h	…
bitops_64.h	sparc: Use popc when possible for ffs/__ffs/ffz.	2011-08-02 21:28:53 -07:00
btext.h	…
bug.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
bugs.h	…
cache.h	sparc32: Remove completely unused code from asm/cache.h	2012-05-19 21:15:37 -07:00
cacheflush.h	sparc32: use flushi when run-time patching in per_cpu_patch	2012-05-20 13:33:36 -07:00
cacheflush_32.h	sparc32: Move cache and TLB flushes over to method ops.	2012-05-13 20:49:31 -07:00
cacheflush_64.h	sparc32: use flushi when run-time patching in per_cpu_patch	2012-05-20 13:33:36 -07:00
cachetlb_32.h	sparc32: Move cache and TLB flushes over to method ops.	2012-05-13 20:49:31 -07:00
chafsr.h	…
checksum.h	…
checksum_32.h	…
checksum_64.h	…
chmctrl.h	…
clock.h	…
cmpxchg.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
cmpxchg_32.h	sparc32: remove runtime btfix support	2012-05-14 14:05:09 -07:00
cmpxchg_64.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
compat.h	sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads.	2012-10-26 15:18:37 -07:00
compat_signal.h	sparc: switch to generic sigaltstack	2013-02-03 18:16:17 -05:00
contregs.h	sparc32: Remove sun4{,c} control reg definitions from contregs.h.	2012-05-11 21:29:17 -07:00
cpu_type.h	sparc32: Remove sun4 and sun4c from enum sparc_cpu.	2012-05-12 00:23:23 -07:00
cpudata.h	…
cpudata_32.h	sparc32: generic clockevent support	2012-04-15 10:28:50 -07:00
cpudata_64.h	…
current.h	…
dcr.h	…
dcu.h	…
delay.h	…
delay_32.h	…
delay_64.h	…
device.h	…
dma-mapping.h	sparc: dma-mapping: support debug_dma_mapping_error	2012-11-17 13:20:38 +01:00
dma.h	sparc32: Convert mmu_* interfaces from btfixup to method ops.	2012-05-13 13:57:05 -07:00
ebus_dma.h	…
ecc.h	…
eeprom.h	…
elf.h	…
elf_32.h	default SET_PERSONALITY() in linux/elf.h	2013-02-26 02:46:08 -05:00
elf_64.h	sparc64: Add detection for features new in SPARC-T4.	2012-08-18 23:02:36 -07:00
estate.h	…
fb.h	…
fbio.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
fhc.h	…
floppy.h	…
floppy_32.h	sparc: Kill mmu_{un,}lockarea().	2012-05-13 13:23:16 -07:00
floppy_64.h	sparc64: Remove pointless assignment in floppy support.	2012-05-14 20:59:25 -07:00
fpumacro.h	…
ftrace.h	…
futex.h	…
futex_32.h	…
futex_64.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
gpio.h	gpiolib/arches: Centralise bolierplate asm/gpio.h	2012-05-11 18:00:14 -06:00
hardirq.h	…
hardirq_32.h	…
hardirq_64.h	…
head.h	…
head_32.h	sparc32: Kill unused defines from asm/head_32.h	2012-05-11 21:43:11 -07:00
head_64.h	…
highmem.h	sparc32: move kmap_init() to highmem.c	2012-07-26 16:46:17 -07:00
hugetlb.h	sparc64: Fix tsb_grow() in atomic context.	2013-02-20 09:46:08 -08:00
hvtramp.h	…
hw_irq.h	…
hypervisor.h	sparc64: Add hypervisor interfaces for SPARC-T4 perf counter access.	2012-08-18 23:03:53 -07:00
ide.h	…
idprom.h	…
intr_queue.h	…
io-unit.h	…
io.h	sparc: add {read,write}*_be routines	2011-05-12 13:44:29 -07:00
io_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
io_64.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
ioctls.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
iommu.h	…
iommu_32.h	…
iommu_64.h	…
irq.h	…
irq_32.h	…
irq_64.h	nmi watchdog: do not use cpp symbol in Kconfig	2012-03-23 16:58:31 -07:00
irqflags.h	…
irqflags_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
irqflags_64.h	sparc/irqs: Do not trace arch_local_{*,irq_*} functions	2011-07-06 08:00:29 -07:00
jump_label.h	static keys: Introduce 'struct static_key', static_key_true()/false() and static_key_slow_[inc|dec]()	2012-02-24 10:05:59 +01:00
kdebug.h	…
kdebug_32.h	…
kdebug_64.h	…
kgdb.h	…
kmap_types.h	…
kprobes.h	…
ldc.h	…
leon.h	sparc32,leon: drop leon_init()	2012-07-26 16:46:16 -07:00
leon_amba.h	sparc32,leon: always include leon_kernel in build	2012-05-27 23:52:46 -07:00
leon_pci.h	PCI: Make the struct pci_dev * argument of pci_fixup_irqs const.	2011-07-22 08:26:06 -07:00
linkage.h	…
lsu.h	…
machines.h	sparc32: Trivial removal of sun4c references in comments.	2012-05-12 00:35:45 -07:00
mbus.h	sparc32: Remove cypress cpu support.	2012-05-15 10:22:00 -07:00
mc146818rtc.h	…
mc146818rtc_32.h	…
mc146818rtc_64.h	…
mdesc.h	sparc64: Probe cpu page size support more portably.	2012-09-06 19:01:25 -07:00
memctrl.h	…
mman.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
mmu.h	…
mmu_32.h	…
mmu_64.h	sparc64: Support transparent huge pages.	2012-10-09 16:23:06 +09:00
mmu_context.h	…
mmu_context_32.h	sparc32: centralize all mmu context handling in srmmu.c	2012-07-26 16:46:16 -07:00
mmu_context_64.h	sparc64: Support transparent huge pages.	2012-10-09 16:23:06 +09:00
mmzone.h	Fix node_start/end_pfn() definition for mm/page_cgroup.c	2011-06-27 14:13:09 -07:00
msi.h	…
mxcc.h	…
nmi.h	…
ns87303.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
obio.h	sparc32: move smp ipi to method ops	2012-05-14 14:05:07 -07:00
openprom.h	…
oplib.h	…
oplib_32.h	sparc: fix format string argument for prom_printf()	2012-10-02 23:20:34 -04:00
oplib_64.h	sparc: fix format string argument for prom_printf()	2012-10-02 23:20:34 -04:00
page.h	…
page_32.h	sparc32: drop sparc_unmapped_base	2012-07-26 16:46:16 -07:00
page_64.h	sparc64: Fix tsb_grow() in atomic context.	2013-02-20 09:46:08 -08:00
parport.h	SPARC: drivers: remove __dev* attributes.	2013-01-03 15:57:04 -08:00
pbm.h	…
pci.h	…
pci_32.h	PCI: collapse pcibios_resource_to_bus	2012-02-23 20:19:04 -07:00
pci_64.h	PCI: collapse pcibios_resource_to_bus	2012-02-23 20:19:04 -07:00
pcic.h	sparc32: added CONFIG_PCIC_PCI Kconfig setting	2011-06-02 14:32:37 -07:00
pcr.h	sparc64: Add PCR ops for SPARC-T4.	2012-08-18 23:26:19 -07:00
percpu.h	…
percpu_32.h	…
percpu_64.h	…
perf_event.h	…
pgalloc.h	…
pgalloc_32.h	sparc32: drop quicklist	2012-07-26 16:46:16 -07:00
pgalloc_64.h	sparc64: Support transparent huge pages.	2012-10-09 16:23:06 +09:00
pgtable.h	…
pgtable_32.h	sparc32: drop unused BAD_PAGE stuff	2012-07-26 16:46:16 -07:00
pgtable_64.h	sparc64: Fix race in TLB batch processing.	2013-04-19 17:26:26 -04:00
pgtsrmmu.h	sparc32: srmmu_probe now knows about leon too	2012-05-27 23:52:51 -07:00
pil.h	…
processor.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
processor_32.h	sparc idle: rename pm_idle to sparc_idle	2013-02-17 23:36:56 -05:00
processor_64.h	Merge branch 'arch-microblaze' into no-rebases	2012-11-16 22:28:43 -05:00
prom.h	of/address: sparc: Declare of_iomap as an extern function for sparc again	2012-11-23 22:01:15 +00:00
psr.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
ptrace.h	Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc	2012-11-16 20:49:06 -05:00
ross.h	…
rwsem.h	…
sbi.h	…
scatterlist.h	…
scratchpad.h	…
seccomp.h	…
sections.h	sparc32: add support for run-time patching of leon/sun single instructions	2012-05-27 23:52:43 -07:00
setup.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
sfafsr.h	…
sfp-machine.h	…
sfp-machine_32.h	…
sfp-machine_64.h	…
shmparam.h	…
shmparam_32.h	sparc32: Remove sun4 and sun4c from enum sparc_cpu.	2012-05-12 00:23:23 -07:00
shmparam_64.h	…
sigcontext.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
siginfo.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
signal.h	consolidate kernel-side struct sigaction declarations	2013-02-03 15:09:22 -05:00
smp.h	…
smp_32.h	sparc:cleanup unused code in smp_32.h	2013-03-31 19:29:12 -04:00
smp_64.h	sparc64: Add global PMU register dumping via sysrq.	2012-10-16 09:34:01 -07:00
sparsemem.h	…
spinlock.h	…
spinlock_32.h	sparc32: unbreak arch_write_unlock()	2011-08-15 14:35:19 -07:00
spinlock_64.h	sparc64: remove unnecessary macros from spinlock_64.h	2011-08-15 14:35:18 -07:00
spinlock_types.h	…
spitfire.h	sparc64: correctly recognize SPARC64-X chips	2013-03-11 05:06:27 -07:00
stacktrace.h	…
starfire.h	…
string.h	…
string_32.h	sparc32: Remove inline strncmp "optimization" for constant counts.	2012-05-11 19:53:29 -07:00
string_64.h	…
sunbpp.h	…
swift.h	…
switch_to.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
switch_to_32.h	Disintegrate asm/system.h for Sparc	2012-03-28 18:30:03 +01:00
switch_to_64.h	sparc64: Fix race in TLB batch processing.	2013-04-19 17:26:26 -04:00
syscall.h	…
syscalls.h	sparc32: switch to generic sys_execve()	2012-10-16 19:03:53 -04:00
termbits.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
termios.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
thread_info.h	…
thread_info_32.h	sanitize tsk_is_polling()	2012-10-01 09:58:13 -04:00
thread_info_64.h	Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc	2012-11-16 20:49:06 -05:00
timer.h	…
timer_32.h	sparc32: remove runtime btfix support	2012-05-14 14:05:09 -07:00
timer_64.h	SPARC: drivers: remove __dev* attributes.	2013-01-03 15:57:04 -08:00
timex.h	…
timex_32.h	sparc32: generic clockevent support	2012-04-15 10:28:50 -07:00
timex_64.h	…
tlb.h	…
tlb_32.h	…
tlb_64.h	sparc: mmu_gather rework	2011-05-25 08:39:13 -07:00
tlbflush.h	…
tlbflush_32.h	sparc32: Move cache and TLB flushes over to method ops.	2012-05-13 20:49:31 -07:00
tlbflush_64.h	sparc64: Fix race in TLB batch processing.	2013-04-19 17:26:26 -04:00
topology.h	…
topology_32.h	…
topology_64.h	sched/numa: Rewrite the CONFIG_NUMA sched domain support	2012-05-09 15:00:55 +02:00
trap_block.h	…
traps.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	2012-10-09 09:47:43 +01:00
tsb.h	sparc64: Fix huge PMD to PTE translation for sun4u in TLB miss handler.	2013-02-20 13:34:41 -08:00
tsunami.h	…
ttable.h	sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads.	2012-10-26 15:18:37 -07:00
turbosparc.h	…
uaccess.h	sparc: Fix user_addr_max() definition.	2012-05-24 13:41:58 -07:00
uaccess_32.h	sparc: use the new generic strnlen_user() function	2012-05-26 11:33:54 -07:00
uaccess_64.h	sparc64: clear syscall_noerror on the entry to syscall, not on the exit	2012-10-14 19:26:52 -04:00
unaligned.h	…
unistd.h	burying unused conditionals	2013-02-14 09:21:15 -05:00
upa.h	…
user.h	…
vaddrs.h	sparc32: drop fixmap.h	2012-07-26 16:46:16 -07:00
vga.h	BUG: headers with BUG/BUG_ON etc. need linux/bug.h	2012-03-04 17:54:34 -05:00
viking.h	…
vio.h	powerpc+sparc/vio: Modernize driver registration	2012-03-28 11:33:24 +11:00
visasm.h	…
winmacro.h	sparc32: Implement hard_smp_processor_id() via instruction patching.	2012-05-14 13:31:38 -07:00
xor.h	…
xor_32.h	…
xor_64.h	sparc64: Future proof Niagara cpu detection.	2011-09-16 14:21:33 -07:00