The format specifier %p can leak kernel addresses
while not valuing the kptr_restrict system settings.
Use %pK instead of %p, which evaluates whether
kptr_restrict is set.
Change-Id: I0778e43e0a03852ca2944377256a7b401586a747
Signed-off-by: Divya Ponnusamy <pdivya@codeaurora.org>
Signed-off-by: Sudeep Yedalapure <sudeepy@codeaurora.org>
The old GPU event model was pretty heavy weight. It walked a lot
of issues while holding the mutex and was generally not very
flexible or scalable.
The new design introduces event groups. Event groups have a list of
common events with a local lock. Each context has a default event
group and the device adds a few more for global events and IOMMU events.
Each event group is processed when timestamps expire and the expired
events are scheduled on a workqueue. We never need to take the device
mutex in any of the event code and we have the flexibility to deal with
both global and per-context timestamps in a reasonably generic manner.
Speed is a premimum at every step of the process, so we use RCUs for
the global group list, a new kmem cache for the event structs and
generally go out of our way to avoid blocking in rendering threads
even a little bit.
Change-Id: Ic0dedbad02eec7c75258e6787543b79d4b8b3394
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Sunil Khatri <sunilkh@codeaurora.org>
Tighten up the memory free list code and remove the memory unused
debugfs file. Outside of the cleanup the functional changes are
a) move the mutex to a spinlock for speed, and b) only return the
last entry in the list for given pid/gpu address combination.
Conflicts:
drivers/gpu/msm/kgsl.c
drivers/gpu/msm/kgsl.h
drivers/gpu/msm/kgsl_debugfs.c
Change-Id: Ic0dedbad9595302e2b42de2807fa82c425d1717f
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Hareesh Gundu <hareeshg@codeaurora.org>
Put CP_STATE_DEBUG_INDEX and CP_STATE_DEBUG_DATA under protection
to keep it from being written from an IB1. Doing so however opens
up a subtle "feature" in the microcode: memory read opcodes turn off
protected mode in the microcode to do the read and then turns it
back on regardless of the initial state. This is a problem if the
memory read happens while protected mode is turned off and then we
try to access a protected register which then complains and goes boom.
To account for this irregularity explicitly turn back off protected
mode in all the places where we know this will be a problem.
Change-Id: Ic0dedbad1397ca9b80132241ac006560a615e042
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Flush TLB when mapping to IOMMU pagetable, this is a requirement
for current IOMMU hardware.
Change-Id: I82e766f4869c737c1f5b73934eacaada40d46e10
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Add new API's kgsl_mutex_lock/unlock to track device mutex owner.
Mutex owner is an atomic variable and all operations set/clear on
it are also atomic to avoid any race conditions.
If a thread tries to acquire the mutex using new API, try to acquire
the mutex only if mutex owner is not same as current thread. This is
to prevent deadlocks which can happen if a thread tries to acquire
mutex with mutex held.
Change-Id: I3da4d0427ede5458f6a663043c57a4799b72d8fd
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Iommu driver could be allocating a valid GPU pagetable at
physical address 0x0, remove assumption from GPU kernel driver
that 0x0 is invalid pagetable address.
Change-Id: I213c0fbc1d90a13bef83c7d7589ad9e5729771fd
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
IOMMU clock disable on timestamp allocates memory and passes
the pointer to event created, this memory should be freed
when the event fires.
Change-Id: Ia714550ce8a05958fb6668293cd0400a0f69d4af
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Allow each mmu clock enable call to enable the clock and let the
clock driver refcounting mechanism deal with switching the clock
off at the right time. This also means that each enable call should
have a disable call or a disable event registered, so allow for
multiple disable events to be registered. This simplifies the mmu
clock handling mechanism which is better for multiple ringbuffer
implementation. With multiple ringbuffers each global timestamp is
tied to a specific ringbuffer and with a single clock disable event
it is hard to figure out when the clock should be turned off.
CRs-fixed: 597193 592485
Change-Id: I924c545140bd19d95890a5fdfd0382c511abe973
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
When changing the pagefault policy the MMU setting for the pagefault
policy was not changed, add code to do that.
Conflicts:
drivers/gpu/msm/adreno.c
Change-Id: I3b185a7f6bdda42856c777c60ae82c32b9c52eb9
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Hareesh Gundu <hareeshg@codeaurora.org>
The check in the code will always evaluate to true now that we
don't support A-family and does not affect the if statement that
uses it.
Change-Id: Ifdf88cc0305eb64a6a50f72a23e69239173ee4c6
Signed-off-by: Shrenuj Bansal <shrenujb@codeaurora.org>
Flushes the entire tlb when calling mmu_unmap to make sure there
is no race condition between unmapping and freeing the pages.
Change-Id: I911f8479a3aef2006f9df40a3419284423937542
Signed-off-by: Shrenuj Bansal <shrenujb@codeaurora.org>
Enable the IOMMU halt feature for non v0 IOMMU versions instead of
depending on the feature being enabled in the IOMMU device tree.
This is because on certain chipsets this feature is not set in
the device tree even if the IOMMU h/w supports it.
Change-Id: I39236b5f7a7dc1d0d739bbb39bfc7fda4931da56
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Add dispatcher and server-side-sync support from the msm-3.10 branch
featuring these commits:
msm: kgsl: Add a command dispatcher to manage the ringbuffer
msm: kgsl: Implement KGSL fault tolerance policy in the dispatcher
msm: kgsl: Don't process events if the timestamp hasn't changed
msm: kgsl: Add a new command submission API
msm: kgsl: add kgsl_sync_fence_waiter for server side sync
msm: kgsl: Add support for KGSL_CMD_SYNCPOINT_TYPE_FENCE
msm: kgsl: Add the event callback symbol to GPU event tracepoints
msm: kgsl: Add a ftrace event for adreno context switches
msm: kgsl: Try to run soft reset on all targets that support it
msm: kgsl: Check for idle in the fault detection timer
msm: kgsl: Improve the dispatcher issuecmds loop
kref: Implement kref_get_unless_zero v3
msm: kgsl: fix a race condition when calling kref_put
msm: kgsl: Check the return value of _kgsl_context_get()
msm: kgsl: Hold the plist spinlock to protect nodes
msm: kgsl: Hold a reference to process private when creating context
msm: kgsl: Do not delete command batch unless it's retired
msm: kgsl: Check whether context is detached before updating timestamp
msm: kgsl: Free the cmdbatch when it fails to be requeued
msm: kgsl: Clear dispatcher fault bit if inflight commands are 0
msm: kgsl: Do not register additional IOMMU faults
msm: kgsl: Do not overwrite dispatcher fault
msm: kgsl: Define new fault type for IOMMU fault
msm: kgsl: On IOMMU fault force a hard reset
msm: kgsl: Max the expired timestamp of invalidated context
msm: kgsl: Do not allow events with bogus timestamp
msm: kgsl: Immediately set the fault bit on a pagefault
msm: kgsl: Set FSR to 0 on IOMMU pagefault
msm: kgsl: Do not call dispatcher fault handler if already called
msm: kgsl: Only skip IB for faulting context
msm: kgsl: Retry hard reset more than once on failure
msm: kgsl: Wake up threads after context detach
msm: kgsl: Don't pause the dispatcher in adreno_dispatcher_stop
msm: kgsl: Fix draw context fair scheduling in the dispatcher
msm: kgsl: Return -EPROTO in certain functions if the GPU has faulted
msm: kgsl: GPU pagefault policy change to not stall on pagefault
msm: kgsl: A few quick cleanups to improve stability
msm: kgsl: Return -ENOTTY for legacy waittimestamp calls
msm: kgsl Check NULL return from kgsl_get_device_from_dev
msm: kgsl: use kgsl_context_detached() in adreno_dispatcher_queue_cmd()
msm: kgsl: remove KGSL_STATE_DUMP_AND_FT
msm: kgsl: Re-add the irq_last functionality
msm: kgsl: Allow sync points to be created on the submitter's context
msm: kgsl: Don't repeatedly call adreno_stop during reset
msm: kgsl: Add an helper function to clear the GPU fault after reset
Change-Id: Ic0dedbad5ae48dddad1e5f76875b0a8604017cc3
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
In cases of page fault, resuming the MMU before detaching
the domain locks up the MMU and causes erroneous behavior.
To avoid this, only resume the MMU after detaching the domain.
Additionally, do not attempt soft reset in cases of page faults.
CRs-fixed: 550840
Change-Id: I0c8ea79d802d8b8d29e03af6f3153ee6c3401400
Signed-off-by: Harsh Vardhan Dwivedi <hdwivedi@codeaurora.org>
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
On IOMMU pagefault the FSR is not cleared until the IOMMU
is reset and re-attached to the GPU device. Until FSR clears
the system will keep generating fault interrupts. Clear the
FSR when iommu is stopped in kgsl driver to prevent system
from generating additional fault interrupts for the same
fault until the IOMMU is re-attached
Change-Id: Ia56bbd1245a84a001ec8464f73cfedf2a2d312c2
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Enable and use GPU hang interrupt to detect GPU hangs.
If the GPU hang is not detected using the interrupt we
still have the hang timer to detect GPU hangs. When
hang is detected using GPU hang interrupt fault tolerance
is triggered. Fault tolerance time is less than 50ms with
GPU hang interrupt.
Conflicts:
drivers/gpu/msm/adreno.h
drivers/gpu/msm/adreno_a2xx.c
drivers/gpu/msm/adreno_a3xx.c
drivers/gpu/msm/adreno_ringbuffer.c
CRs-Fixed: 512795
Change-Id: I9327803f178ae1b5e8248fe3450499b7bba2a68d
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
The remote spinlock was originally meant to synchronize access between
GPU and CPU accessing SMMU. However, there are more use cases that
this remote spinlock serves. This change decouples the spinlock from
GPU and adds support for taking the spinlock only for SMMUs that are
specified in the device tree.
CRs-fixed: 517873
Change-Id: Ic50992d0d1a102fbd05855e09e254e627f99ec33
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Implement the KGSL fault tolerance policy for faults in the dispatcher.
Replay (or skip) the inflight command batches as dictated by the policy,
iterating progressively through the various behaviors.
Change-Id: Ic0dedbade98cc3aa35b26813caf4265c74ccab56
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Implements a centralized dispatcher for sending user commands
to the ringbuffer. Incoming commands are queued by context and
sent to the hardware on a round robin basis ensuring each context
a small burst of commands at a time. Each command is tracked
throughout the pipeline giving the dispatcher better knowledge
of how the hardware is being used. This will be the basis for
future per-context and cross context enhancements as priority
queuing and server-side syncronization.
Change-Id: Ic0dedbad49a43e8e6096d1362829c800266c2de3
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Add a type parameter to event callbacks to give tasks a way to
communicate error conditions to the callback. Current type values
are KGSL_EVENT_TIMESTAMP_RETIRED for normal timestamp expiry and
KGSL_EVENT_CANCELLED if the event is cancelled prematurely.
Change-Id: Ic0dedbad9907cd50642a604a3af562e01a4b4a3b
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
IOMMU get context functions return error as a pointer so just
checking for NULL is not correct. Use the right macro to check
if error is returned and only force initialization of one context
for IOMMU-v1 since other contexts are not used.
Change-Id: I5123363567649568127e24738f19ea30f98a6cec
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Fix the addressing of FSYNR0 and FSYNR1 IOMMU-v1 registers. These
are context registers and were being addressed as global registers.
Change-Id: I2f6c4798a3c82bb4857a334beb99994ac9f4a1e8
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Having a separate allocated struct for the device specific context
makes ownership unclear, which could lead to reference counting
problems or invalid pointers. Also, duplicate members were
starting to appear in adreno_context because there wasn't a safe
way to reach the kgsl_context from some parts of the adreno code.
This can now be done via container_of().
This change alters the lifecycle of the context->id, which is
now freed when the context reference count hits zero rather
than in kgsl_context_detach().
It also changes the context creation and destruction sequence.
The device specific code must allocate a structure containing
a struct kgsl_context and passes a pointer it to kgsl_init_context()
before doing any device specific initialization. There is also a
separate drawctxt_detach() callback for doing device specific
cleanup. This is separate from freeing memory, which is done
by the drawctxt_destroy() callback.
Change-Id: I7d238476a3bfec98fd8dbc28971cf3187a81dac2
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
There are cases where GPU has a pagefault and executes
fine after pagefault without GPU stall. init.qcom.graphics.sh
script can be used to change FT pagefault policy
to not stall IOMMU V1 on pagefault and check if pagefault is harmless.
Change-Id: If061230b66181bfd94c697ea106e7bf4de352e91
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Everybody loves a rcu but in this case we are dangerously mixing rcus and
atomic operations. Add a read/write lock to explicitly protect the idr.
Also fix a few spots where the idr was used without protection.
Change-Id: Ic0dedbad517a9f89134cbcf7af29c8bf0f034708
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Add a memory barrier after writing to V2PUR register of IOMMU. Without
this barrier the CPU core can stall because of multiple outstanding IOMMU
register transactions happening in parallel. This barrier synchronizes
the multiple IOMMU register transactions and prevents the CPU from
stalling.
Change-Id: I1d715e9fb8287b1f0b009b39b11b5469ab8861c8
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
in check_if_freed() a call to kgsl_get_memory_usage() was terminated with
a comma instead of a semicolon. Through a quirk in a macro this somehow
managed to compile and function properly but that doesn't make it okay.
CRs-fixed: 497280
Change-Id: Ic0dedbadfdf901bbc68b7f8cefb231d08dd01a95
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Soft reset GPU when GPU power rail is not turned off.
In hard reset we reload full microcode but in soft reset
since the memory powerrail is not turned off we only
load jump tables part of microcode, this reduces GPU
reset time from 10ms to 200us.
Change-Id: Ibbc36e97cc95425e13856fd5d847eed742743723
Signed-off-by: Tarun Karra <tkarra@codeaurora.org>
Modify the code and update makefile and defconfig to allow us to
only compile one of the two IOMMU drivers that we have. Only one of
the drivers are needed at the same time. This will ensure we are not
compiling and linking code that is not needed.
Change-Id: I7db69f873245e57bddab6625e0b04e2ed48f44b6
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Future chipsets may not define phys_addr_t to 32 bits hence convert
all physical address variables to this type.
Change-Id: I4ac5bd1aabda455456ff867c973a264f68992404
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Make CFF capture a device specific property. This allows the control
of CFF for a particular device without CFF interferance from another
device. This will be useful when we have a virtual device and need to
only capture CFF for the virtual device. CFF capture can only be
turned on for one device at a time.
Change-Id: I14c5a4442ad05327de1413d98bf795dbd196119d
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
If remote spin lock is not configured to be used then the
initialization function will return a NULL value. Check this
value before using it.
Change-Id: Iacb4b7914a93e7aefcd20341ecedbaa466c98acb
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Y-T-G
Francescodario Cuzzocrea
Divya Ponnusamy
Sunil Khatri
Jordan Crouse
Tarun Karra
Linux Build Service Account
Shubhraprakash Das
Shrenuj Bansal
Zhoulu Luo
Harsh Vardhan Dwivedi
Olav Haugan
Jeremy Gebben