Skip to content

perf(file-list): db pagination - #17505

Open
alperozturk96 wants to merge 1 commit into
masterfrom
fix/large-folder-listing
Open

perf(file-list): db pagination#17505
alperozturk96 wants to merge 1 commit into
masterfrom
fix/large-folder-listing

Conversation

@alperozturk96

Copy link
Copy Markdown
Collaborator

Issue

When user have a lot of files and/or folder in directory app crashes. Due to fetching all files at once from DB.

{
      "header": {
        "logLevel": "ERROR",
        "pid": 7829,
        "tid": 7849,
        "applicationId": "com.nextcloud.client",
        "processName": "com.nextcloud.client",
        "tag": "CursorWindow",
        "timestamp": {
          "seconds": 1787142876,
          "nanos": 504903577
        }
      },
      "message": "Failed to read row 615, column 5 from a window with 615 rows, 52 columns"
},

Changes

  • Fetch batch by batch. Batch size is 100.
  • Return ID from DB instead of FileEntity.

How to reproduce crash?

  1. Have 2K files and 2K folders in root directory.
  2. Keep scroll
  3. Crash

Signed-off-by: alperozturk96 <alper_ozturk@proton.me>
@alperozturk96 alperozturk96 added this to the Nextcloud App 35.0.0 milestone Aug 19, 2026
@alperozturk96 alperozturk96 added 3. to review performance 馃殌 Performance improvement opportunities (non-crash related) labels Aug 19, 2026
@github-actions

Copy link
Copy Markdown

blue-Light-Screenshot test failed, but no output was generated. Maybe a preliminary stage failed.

@github-actions

Copy link
Copy Markdown

Codacy

SpotBugs

CategoryBaseNew
Bad practice3232
Correctness5959
Dodgy code207207
Experimental11
Malicious code vulnerability22
Multithreaded correctness2525
Performance3939
Security1919
Total384384

@github-actions

Copy link
Copy Markdown

APK file: https://github.com/nextcloud/android/actions/runs/32258040777/artifacts/9367607142
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
qrcode (please click on link to get QR code displayed)

@daniele-verducci daniele-verducci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Alper,
I tried it, but it's still crashing for me. Here's the stacktrace:

2026-08-24 15:25:28.892 12717-12765 ReadFolder...eOperation com.nextcloud.android.qa             E  Synchronized /: failed
2026-08-24 15:25:28.893 12717-12743 WM-WorkerWrapper        com.nextcloud.android.qa             E  Work [ id=d7d541fe-9e26-4998-9176-87424ecf4e04, tags={ com.nextcloud.client.jobs.metadata.MetadataWorker,*,name:metadata_sync,timestamp:1787577910001,class:MetadataWorker } ] failed because it threw an exception/error
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 16 byte allocation with 2295520 free bytes and 2241KB until OOM, target footprint 603979776, growth limit 603979776; giving up on allocation because <1% of heap free after GC.
                                                                                                    	at org.apache.harmony.xml.dom.InnerNodeImpl.getChildNodes(InnerNodeImpl.java:56)
                                                                                                    	at org.apache.jackrabbit.webdav.xml.DomUtil.getContent(DomUtil.java:397)
                                                                                                    	at org.apache.jackrabbit.webdav.property.DefaultDavProperty.createFromXml(DefaultDavProperty.java:130)
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatusResponse.createFromXml(MultiStatusResponse.java:463)
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatus.createFromXml(MultiStatus.java:185)
                                                                                                    	at org.apache.jackrabbit.webdav.client.methods.DavMethodBase.processResponseBody(DavMethodBase.java:282)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1749)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
                                                                                                    	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
                                                                                                    	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
                                                                                                    	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
                                                                                                    	at com.owncloud.android.lib.common.OwnCloudClient.executeMethod(OwnCloudClient.java:192)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.run(ReadFolderRemoteOperation.java:62)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:193)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.fetchAndSyncRemoteFolder(RefreshFolderOperation.java:466)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.run(RefreshFolderOperation.java:281)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:132)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:141)
                                                                                                    	at com.nextcloud.client.jobs.metadata.MetadataWorker$refreshFolder$2.invokeSuspend(MetadataWorker.kt:108)
                                                                                                    	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
                                                                                                    	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
                                                                                                    	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
                                                                                                    	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
2026-08-24 15:25:34.018 12717-13036 ReadFolder...eOperation com.nextcloud.android.qa             E  Synchronized /: failed
2026-08-24 15:25:34.018 12717-13036 AndroidRuntime          com.nextcloud.android.qa             E  FATAL EXCEPTION: Thread-17
                                                                                                    Process: com.nextcloud.android.qa, PID: 12717
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 80032 byte allocation with 968416 free bytes and 945KB until OOM, target footprint 603979776, growth limit 603979776; giving up on allocation because <1% of heap free after GC.
                                                                                                    	at org.apache.jackrabbit.webdav.MultiStatus.getResponses(MultiStatus.java:128)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.readData(ReadFolderRemoteOperation.java:133)
                                                                                                    	at com.owncloud.android.lib.resources.files.ReadFolderRemoteOperation.run(ReadFolderRemoteOperation.java:70)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:193)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.fetchAndSyncRemoteFolder(RefreshFolderOperation.java:466)
                                                                                                    	at com.owncloud.android.operations.RefreshFolderOperation.run(RefreshFolderOperation.java:281)
                                                                                                    	at com.owncloud.android.lib.common.operations.RemoteOperation.run(RemoteOperation.java:387)
                                                                                                    	at java.lang.Thread.run(Thread.java:1571)

If it's relevant, here's how I tested it:

  • pawned docker container (github.com/tobiasKaminsky/nextcloud-client-testing-server)
  • created empty dirs and ampty files with bash for loop and touch (so the files are 0 bytes .txt files)
  • runned occ files:scan
  • opened in browser to check the APIs are not crashing (OK)
  • installed QA APK build from CI and opened: crashes after a while

@alperozturk96 alperozturk96 mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review performance 馃殌 Performance improvement opportunities (non-crash related)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants