Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: 6 GBs files (307 chunks) wont join #681

Open
audricd opened this issue Jan 31, 2025 · 11 comments
Open

Bug: 6 GBs files (307 chunks) wont join #681

audricd opened this issue Jan 31, 2025 · 11 comments
Labels
bug Something isn't working

Comments

@audricd
Copy link

audricd commented Jan 31, 2025

What happened?

Hello,

So I was uploading a 6gbs file, 307 chunks. Once the upload was complete, I got the usual pop up of offering me a link while I wait the background process finishes.

However. It seems that background process wont finish, it wont even start.

Image

File number 2 by the way, is a previous attempt of the same file.
Its been now, close to an hour. Same for previous file (before I deleted it) in which, the status remains on "pending".

The log dont tell much.

On top of deleting the previous upload, and reuploading it, I have restarted the containers stack twice. The status havent changed whatsoever.

I am running on 3.7.11

Version

latest (ghcr.io/diced/zipline or ghcr.io/diced/zipline:latest)

What browser(s) are you seeing the problem on?

No response

Zipline Logs

2025-01-30 11:35:30,347 AM info  [datasource] using S3(bakaps) datasource
2025-01-30 11:35:30,402 AM info  [database::migrations] establishing database connection
2025-01-30 11:35:30,403 AM info  [database::migrations] ensuring database exists, if not creating database - may error if no permissions
2025-01-30 11:35:30,650 AM info  [database::migrations] exiting migrations engine - database is up to date
2025-01-30 11:35:32,507 AM info  [server] listening on 0.0.0.0:3000
2025-01-30 11:35:32,507 AM info  [server] started production [email protected] server
2025-01-31 11:31:31,581 AM info  [login] User RpW6d3nXUgHAtyrDX01G (1) logged in
2025-01-31 11:32:07,153 AM info  [folders] Deleted folder "media" for user RpW6d3nXUgHAtyrDX01G (1)
2025-01-31 11:32:09,860 AM info  [folders] Deleted folder "juegos" for user RpW6d3nXUgHAtyrDX01G (1)
2025-01-31 11:32:13,177 AM info  [folders] Deleted folder "ost" for user RpW6d3nXUgHAtyrDX01G (1)
2025-01-31 11:52:59,323 AM info  [upload] User RpW6d3nXUgHAtyrDX01G (1) uploaded ZWUYpv.7z (4)
2025-01-31 04:12:48,633 PM info  [datasource] using S3(bakaps) datasource
2025-01-31 04:13:14,557 PM info  [worker::upload::6Unr] User RpW6d3nXUgHAtyrDX01G (1) uploaded EzLkm5.7z (5) (chunked)
2025-01-31 04:23:28,208 PM info  [datasource] using S3(bakaps) datasource
2025-01-31 04:48:37,497 PM info  [files] User RpW6d3nXUgHAtyrDX01G (1) deleted an image zcBsSc.zip (6) owned by RpW6d3nXUgHAtyrDX01G (1)
2025-01-31 04:55:01,821 PM info  [datasource] using S3(bakaps) datasource
2025-01-31 05:03:02,517 PM info  [datasource] using S3(bakaps) datasource
2025-01-31 05:03:02,591 PM error [server] Found temporary files in Zipline's temp directory.
2025-01-31 05:03:02,591 PM error [server] This can happen if Zipline crashes or is stopped while chunking a file.
2025-01-31 05:03:02,591 PM error [server] If you are sure that no files are currently being processed, you can delete the files in the temp directory.
2025-01-31 05:03:02,592 PM error [server] The temp directory is located at: /tmp/zipline
2025-01-31 05:03:02,592 PM error [server] If you are unsure, you can safely ignore this message.
2025-01-31 05:03:02,593 PM info  [database::migrations] establishing database connection
2025-01-31 05:03:02,594 PM info  [database::migrations] ensuring database exists, if not creating database - may error if no permissions
2025-01-31 05:03:02,822 PM info  [database::migrations] exiting migrations engine - database is up to date
2025-01-31 05:03:04,556 PM info  [server] listening on 0.0.0.0:3000
2025-01-31 05:03:04,556 PM info  [server] started production [email protected] server
2025-01-31 05:22:08,867 PM info  [datasource] using S3(bakaps) datasource
2025-01-31 05:22:08,929 PM error [server] Found temporary files in Zipline's temp directory.
2025-01-31 05:22:08,929 PM error [server] This can happen if Zipline crashes or is stopped while chunking a file.
2025-01-31 05:22:08,930 PM error [server] If you are sure that no files are currently being processed, you can delete the files in the temp directory.
2025-01-31 05:22:08,930 PM error [server] The temp directory is located at: /tmp/zipline
2025-01-31 05:22:08,930 PM error [server] If you are unsure, you can safely ignore this message.
2025-01-31 05:22:08,931 PM info  [database::migrations] establishing database connection
2025-01-31 05:22:08,932 PM info  [database::migrations] ensuring database exists, if not creating database - may error if no permissions
2025-01-31 05:22:09,215 PM info  [database::migrations] exiting migrations engine - database is up to date
2025-01-31 05:22:11,062 PM info  [server] listening on 0.0.0.0:3000
2025-01-31 05:22:11,062 PM info  [server] started production [email protected] server

Browser Logs



Additional Info


No response

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @audricd
audricd




          added
  the 

  bug

  Something isn't working
 label


      Jan 31, 2025

    

  








      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 1, 2025


      
    


  





      


        


  
    

      
          
here is my stack:


version: '3'
services:
  postgres:
    image: postgres:15
    restart: unless-stopped
    environment:
      - POSTGRES_USER=someuser
      - POSTGRES_PASSWORD=somepassword
      - POSTGRES_DATABASE=somedb
    volumes:
      - pg_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U VAjNmzJRCEBXwP0Ewbb9']
      interval: 10s
      timeout: 5s
      retries: 5

  zipline:
    image: ghcr.io/diced/zipline
    restart: unless-stopped

    #ports:
    #  - '4000:3000'
    env_file:
      - stack.env
    environment:
      - CORE_SECRET=somesecret
      - CORE_DATABASE_URL=postgres://someuser:somepassword@postgres/somedb
    volumes:
      - '/home/user/storagebox/uploads:/zipline/uploads'
      - '/home/user/storagebox/public:/zipline/public'
    depends_on:
      - 'postgres'

volumes:
  pg_data:

networks:
  default:
    external:
      name: ext



my stack.env:


MFA_TOTP_ENABLED=true
DATASOURCE_S3_ACCESS_KEY_ID=someid
DATASOURCE_S3_SECRET_ACCESS_KEY=somesecret
DATASOURCE_S3_ENDPOINT=hel1.your-objectstorage.com
DATASOURCE_S3_BUCKET=bucketname
DATASOURCE_S3_FORCE_S3_PATH=false
DATASOURCE_S3_USE_SSL=true
DATASOURCE_TYPE=s3



here is my storagebox tree:


Image


Here is my bucket content:


Image


partial files are found in /tmp/zipline:


Image

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @diced





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Owner


      

  


  

    

      

      
            diced
  

      

      

      commented


        Feb 1, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
could be a memory issue since it has to load each partial into memory (especially for s3). Maybe changing the temporary files location to somewhere on disk, unless you have already done that. I would also turn on debug logs.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 2, 2025


      
    


  





      


        


  
    

      
          
I see


Well, I am doing both things (changing location to drive instead of storage box, and enabling debug log):


2025-02-02 12:34:36,284 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5536481280_5557452800
2025-02-02 12:34:39,691 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5557452800,"end":5578424320,"total":6423585774}
2025-02-02 12:34:39,691 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5557452800_5578424320
2025-02-02 12:34:42,986 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5578424320,"end":5599395840,"total":6423585774}
2025-02-02 12:34:42,986 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5578424320_5599395840
2025-02-02 12:34:47,005 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5599395840,"end":5620367360,"total":6423585774}
2025-02-02 12:34:47,005 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5599395840_5620367360
2025-02-02 12:34:50,275 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5620367360,"end":5641338880,"total":6423585774}
2025-02-02 12:34:50,275 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5620367360_5641338880
2025-02-02 12:34:53,731 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5641338880,"end":5662310400,"total":6423585774}
2025-02-02 12:34:53,731 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5641338880_5662310400
2025-02-02 12:34:56,975 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5662310400,"end":5683281920,"total":6423585774}
2025-02-02 12:34:56,976 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5662310400_5683281920
2025-02-02 12:35:00,475 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5683281920,"end":5704253440,"total":6423585774}
2025-02-02 12:35:00,476 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5683281920_5704253440
2025-02-02 12:35:03,712 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5704253440,"end":5725224960,"total":6423585774}
2025-02-02 12:35:03,712 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5704253440_5725224960
2025-02-02 12:35:07,280 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5725224960,"end":5746196480,"total":6423585774}
2025-02-02 12:35:07,280 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5725224960_5746196480
2025-02-02 12:35:10,508 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5746196480,"end":5767168000,"total":6423585774}
2025-02-02 12:35:10,509 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5746196480_5767168000
2025-02-02 12:35:13,923 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5767168000,"end":5788139520,"total":6423585774}
2025-02-02 12:35:13,923 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5767168000_5788139520
2025-02-02 12:35:17,151 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5788139520,"end":5809111040,"total":6423585774}
2025-02-02 12:35:17,152 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5788139520_5809111040
2025-02-02 12:35:20,376 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5809111040,"end":5830082560,"total":6423585774}
2025-02-02 12:35:20,376 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5809111040_5830082560
2025-02-02 12:35:23,658 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5830082560,"end":5851054080,"total":6423585774}
2025-02-02 12:35:23,658 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5830082560_5851054080
2025-02-02 12:35:27,075 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5851054080,"end":5872025600,"total":6423585774}
2025-02-02 12:35:27,075 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5851054080_5872025600
2025-02-02 12:35:30,350 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5872025600,"end":5892997120,"total":6423585774}
2025-02-02 12:35:30,351 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5872025600_5892997120
2025-02-02 12:35:33,727 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5892997120,"end":5913968640,"total":6423585774}
2025-02-02 12:35:33,727 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5892997120_5913968640
2025-02-02 12:35:37,325 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5913968640,"end":5934940160,"total":6423585774}
2025-02-02 12:35:37,325 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5913968640_5934940160
2025-02-02 12:35:40,787 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5934940160,"end":5955911680,"total":6423585774}
2025-02-02 12:35:40,788 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5934940160_5955911680
2025-02-02 12:35:44,090 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5955911680,"end":5976883200,"total":6423585774}
2025-02-02 12:35:44,090 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5955911680_5976883200
2025-02-02 12:35:47,467 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5976883200,"end":5997854720,"total":6423585774}
2025-02-02 12:35:47,468 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5976883200_5997854720
2025-02-02 12:35:50,659 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":5997854720,"end":6018826240,"total":6423585774}
2025-02-02 12:35:50,660 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_5997854720_6018826240
2025-02-02 12:35:53,981 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6018826240,"end":6039797760,"total":6423585774}
2025-02-02 12:35:53,982 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6018826240_6039797760
2025-02-02 12:35:57,329 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6039797760,"end":6060769280,"total":6423585774}
2025-02-02 12:35:57,329 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6039797760_6060769280
2025-02-02 12:36:00,577 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6060769280,"end":6081740800,"total":6423585774}
2025-02-02 12:36:00,577 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6060769280_6081740800
2025-02-02 12:36:04,037 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6081740800,"end":6102712320,"total":6423585774}
2025-02-02 12:36:04,038 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6081740800_6102712320
2025-02-02 12:36:07,531 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6102712320,"end":6123683840,"total":6423585774}
2025-02-02 12:36:07,531 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6102712320_6123683840
2025-02-02 12:36:10,866 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6123683840,"end":6144655360,"total":6423585774}
2025-02-02 12:36:10,867 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6123683840_6144655360
2025-02-02 12:36:14,282 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6144655360,"end":6165626880,"total":6423585774}
2025-02-02 12:36:14,282 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6144655360_6165626880
2025-02-02 12:36:17,617 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6165626880,"end":6186598400,"total":6423585774}
2025-02-02 12:36:17,617 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6165626880_6186598400
2025-02-02 12:36:21,080 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6186598400,"end":6207569920,"total":6423585774}
2025-02-02 12:36:21,080 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6186598400_6207569920
2025-02-02 12:36:24,339 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6207569920,"end":6228541440,"total":6423585774}
2025-02-02 12:36:24,339 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6207569920_6228541440
2025-02-02 12:36:27,769 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6228541440,"end":6249512960,"total":6423585774}
2025-02-02 12:36:27,769 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6228541440_6249512960
2025-02-02 12:36:31,065 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6249512960,"end":6270484480,"total":6423585774}
2025-02-02 12:36:31,065 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6249512960_6270484480
2025-02-02 12:36:34,335 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6270484480,"end":6291456000,"total":6423585774}
2025-02-02 12:36:34,335 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6270484480_6291456000
2025-02-02 12:36:37,776 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6291456000,"end":6312427520,"total":6423585774}
2025-02-02 12:36:37,776 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6291456000_6312427520
2025-02-02 12:36:41,269 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6312427520,"end":6333399040,"total":6423585774}
2025-02-02 12:36:41,269 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6312427520_6333399040
2025-02-02 12:36:45,497 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6333399040,"end":6354370560,"total":6423585774}
2025-02-02 12:36:45,497 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6333399040_6354370560
2025-02-02 12:36:48,800 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6354370560,"end":6375342080,"total":6423585774}
2025-02-02 12:36:48,800 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6354370560_6375342080
2025-02-02 12:36:52,339 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6375342080,"end":6396313600,"total":6423585774}
2025-02-02 12:36:52,339 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6375342080_6396313600
2025-02-02 12:36:55,589 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":false,"start":6396313600,"end":6417285120,"total":6423585774}
2025-02-02 12:36:55,590 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6396313600_6417285120
2025-02-02 12:36:56,769 PM debug [upload] recieved partial upload {"filename":"Wolfenstein_Win_EN_Repack.zip","mimetype":"application/x-zip-compressed","identifier":"1cjA","lastchunk":true,"start":6417285120,"end":6438256640,"total":6423585774}
2025-02-02 12:36:56,769 PM debug [upload] writing partial to disk /tmp/zipline/zipline_partial_1cjA_6417285120_6438256640
2025-02-02 12:36:57,454 PM debug [config] attemping to read .env.local/.env or environment variables
2025-02-02 12:36:57,457 PM debug [config] Attemping to validate {"core":{"secret":"somesecret","database_url":"postgresconnectionstring"},"datasource":{"type":"s3","s3":{"access_key_id":"someaccesskeyid","secret_access_key":"someaccesssecret","endpoint":"hel1.your-objectstorage.com","bucket":"somebucket","force_s3_path":false,"use_ssl":true}},"mfa":{"totp_enabled":true}}
2025-02-02 12:36:57,459 PM debug [config] Recieved config: {"core":{"secret":"somesecret","database_url":"postgresconnectionstring","return_https":false,"temp_directory":"/tmp/zipline","host":"0.0.0.0","port":3000,"logger":false,"stats_interval":1800,"invites_interval":1800,"thumbnails_interval":600,"compression":{"enabled":false,"on_dashboard":false}},"datasource":{"type":"s3","local":{"directory":"./uploads"},"s3":{"access_key_id":"someid","secret_access_key":"someaccesskey","endpoint":"hel1.your-objectstorage.com","bucket":"bakaps","force_s3_path":false,"region":"us-east-1","use_ssl":true}},"uploader":{"default_format":"RANDOM","route":"/u","embed_route":"/a","length":6,"admin_limit":104857600,"user_limit":104857600,"disabled_extensions":[],"format_date":"YYYY-MM-DD_HH:mm:ss","default_expiration":null,"assume_mimetypes":false},"urls":{"route":"/go","length":6},"ratelimit":{"user":0,"admin":0},"website":{"title":"Zipline","show_files_per_user":true,"show_version":true,"disable_media_preview":false,"external_links":[{"label":"Zipline","link":"https://github.com/diced/zipline"},{"label":"Documentation","link":"https://zipline.diced.sh/"}]},"discord":null,"oauth":null,"features":{"invites":false,"invites_length":6,"oauth_registration":false,"oauth_login_only":false,"user_registration":false,"headless":false,"default_avatar":null,"robots_txt":false,"thumbnails":false},"chunks":{"max_size":94371840,"chunks_size":20971520,"enabled":true},"mfa":{"totp_issuer":"Zipline","totp_enabled":true},"exif":{"enabled":false,"remove_gps":false},"ssl":null}
2025-02-02 12:36:57,460 PM info  [datasource] using S3(bakaps) datasource
2025-02-02 12:36:57,461 PM debug [worker::upload::1cjA] starting worker
2025-02-02 12:37:04,363 PM debug [server::invites] deleted 0 used invites
2025-02-02 12:37:04,423 PM debug [server] full size: 1003393833
2025-02-02 12:37:04,430 PM debug [server] by user: [{"_count":{"_all":3},"userId":1}]
2025-02-02 12:37:04,435 PM debug [server] count users: 1
2025-02-02 12:37:04,440 PM debug [server] count by user: [{"username":"someusername","count":3}]
2025-02-02 12:37:04,444 PM debug [server] count files: 3
2025-02-02 12:37:04,449 PM debug [server] sum views: {"_sum":{"views":7}}
2025-02-02 12:37:04,453 PM debug [server] types count: [{"_count":{"mimetype":2},"mimetype":"application/x-7z-compressed"},{"_count":{"mimetype":1},"mimetype":"application/x-zip-compressed"}]
2025-02-02 12:37:04,453 PM debug [server] types count: [{"mimetype":"application/x-7z-compressed","count":2},{"mimetype":"application/x-zip-compressed","count":1}]
2025-02-02 12:37:04,460 PM debug [server::stats] stats updated {"size":"956.9 MB","size_num":1003393833,"count":3,"count_by_user":[{"username":"someusername","count":3}],"count_users":1,"views_count":7,"types_count":[{"mimetype":"application/x-7z-compressed","count":2},{"mimetype":"application/x-zip-compressed","count":1}]}



I am a bit confused. After receiving the total of chunks, its not mentionning anything about joining them? just, say the datasource is s3, reminder of connection string, the instance config (invites, registration, etc whatever) but nothing about the actual pending job?


On the UI same as before. The job is pending. 0/307 chunks. Yesterday I left it 15 hours. The server its on, may not be the most powerful on earth, but I think it should be able to handle joining 307 chunks to make 6 GBs whole, its a hetzner CX42. 8 vcpu, 16 GB ram, 160 GB storage, 20 TB transfer.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 2, 2025


      
    


  





      


        


  
    

      
          
I have by the way while implemented the suggested changes, upgraded to 3.7.12


Image


Its been over 7 hours and it remains at the same point


Image


Image

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @TacticalTechJay





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            TacticalTechJay
  

      

      

      commented


        Feb 3, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
The last image looks like logs for stats updating 🤔

Do you have any logs like this:


2025-02-02 11:32:32,922 PM debug [worker::upload::WTwj] starting worker
2025-02-02 11:32:33,010 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,064 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,116 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,164 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,203 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,254 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,292 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,339 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,382 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,426 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,468 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,515 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,557 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,609 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,648 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,694 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,730 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,764 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,800 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,831 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,863 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,877 PM debug [worker::upload::WTwj::bytes] wrote 18743639 bytes to array
2025-02-02 11:32:33,908 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,946 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:33,979 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,018 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,054 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,096 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,134 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,177 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,211 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,255 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,290 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,324 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,356 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,388 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,420 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,453 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,483 PM debug [worker::upload::WTwj::bytes] wrote 52428800 bytes to array
2025-02-02 11:32:34,488 PM debug [worker::upload::WTwj] writing file to datasource


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @TacticalTechJay





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

  





      


        


  
    

      
          
or some snippets that look like this?


2025-02-02 11:32:32,617 PM debug [upload] recieved partial upload {"filename":"Scott Pilgrim vs The World.mkv","mimetype":"video/x-matroska","identifier":"WTwj","lastchunk":true,"start":1992294400,"end":2044723200,"total":2011038039}
2025-02-02 11:32:32,617 PM debug [upload] writing partial to disk /Temp/zipline/zipline_partial_WTwj_1992294400_2044723200
2025-02-02 11:32:32,643 PM info  [server::response] POST /api/upload -> 200


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 3, 2025


      
    


  





      


        


  
    

      
          
nope, I have shared all logs of the operations, and there is nothing alike.

Chunks uploading

Stats update loop


And nothing else.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 3, 2025


      
    


  





      


        


  
    

      
          
I split the file in two and it worked instantly and flawlessly.

I keep the ticket open in case further troubleshooting and debugging is interesting to anyone else. But to me, the problem is "fixed" (workaround). But since I dont plan on uploading many files as big...

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @diced





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Owner


      

  


  

    

      

      
            diced
  

      

      

      commented


        Feb 3, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          


I split the file in two and it worked instantly and flawlessly.

I keep the ticket open in case further troubleshooting and debugging is interesting to anyone else. But to me, the problem is "fixed" (workaround). But since I dont plan on uploading many files as big...




i don't think i'll implement a fix for this in v3, but if the issue persists in v4 it'll most likely get handled over there (i will test this soon with a big file). another question: are you using s3? i guess it's not a memory issue since you have a beefy server lol

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @audricd





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            audricd
  

      

      

      commented


        Feb 3, 2025


      
    


  





      


        


  
    

      
          
Yep that's fine

Yes, hetzner buckets.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @TacticalTechJay





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

  





      


        


  
    

      
          
Select those in progress chunk uploads and clear them, clear your temp directory, delete the file that should've been made whole, and reupload on the latest commit (trunk branch). I changed s3's saving a bit to use multipart uploading (a server side thing, no change to uploading). If the chunk upload still doesn't work, collect any logs that are like this or this after writing partial file to disk logs. Check your browser's console or network logs too while on the dashboard uploading.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    
















  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    bug

  Something isn't working








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




          
    
No branches or pull requests







    
  




      

    

  
      

  

    

      3 participants
    

    

        
          @audricd 
        
          @TacticalTechJay 
        
          @diced