museum crash when replication enabled #4063

fredix · 2024-11-17T10:25:40Z

fredix
Nov 17, 2024

Hi all,
I tested it a few days ago replication between my Scaleway S3 bucket and a local S3 server at home (garage). But museum crashed. BTW museum works perfectly whitout replication with my local S3 server as primary or the scaleway S3 server as primary.

So now I have setup a third local S3 server. I tested with different configuration with are_local_buckets: true or are_local_buckets: false and with or without:

    hot_storage:
        primary: b2-eu-cen
        secondary: wasabi-eu-central-2-v3

but museum crash like before. Here my config files

museum.yaml

apps:
    # Default is https://albums.ente.io
    #
    # If you're running a self hosted instance and wish to serve public links,
    # set this to the URL where your albums web app is running.
    public-albums: https://ente-albums.fredix.xyz

key:
    encryption: ENCRYPT
    hash: HASH


jwt:
    secret: SECRET


s3:
    # Override the primary and secondary hot storage. The commented out values
    # are the defaults.
    #
    hot_storage:
        primary: b2-eu-cen
        secondary: wasabi-eu-central-2-v3
    b2-eu-cen:
        key: KEY
        secret: SECRET
        endpoint: https://s3.fr-par.scw.cloud
        region: fr-par
        bucket: fredix-ente
    wasabi-eu-central-2-v3:
        key: KEY
        secret: SECRET
        endpoint: http://192.168.0.163:3900
        region: garage
        bucket: ente
    scw-eu-fr-v3:
        key: KEY
        secret: SECRET
        endpoint: http://192.168.0.171:3900
        region: garage
        bucket: ente

    are_local_buckets: false
    use_path_style_urls: true


# Replication config
#
# If enabled, replicate each file to 2 other data centers after it gets
# successfully uploaded to the primary hot storage.
replication:
    enabled: true
    # The Cloudflare worker to use to download files from the primary hot
    # bucket. If this isn't specified, files will be downloaded directly.
    worker-url:
    # Number of go routines to spawn for replication
    # This is not related to the worker-url above.
    # Optional, default value is indicated here.
    worker-count: 6
    # Where to store temporary objects during replication v3
    # Optional, default value is indicated here.
    tmp-storage: /replication/data

log crash

INFO[0000]main.go:95 main Booting up local server with commit #b0674cd5f37cebda0f446db7ce1e87a3e6d8b9fc 
INFO[0000]main.go:791 setupDatabase Setting up db                                
INFO[0000]main.go:798 setupDatabase Connected to DB                              
panic: dial tcp [::1]:5432: connect: connection refused

goroutine 1 [running]:
main.setupDatabase()
	/etc/ente/cmd/museum/main.go:801 +0x36b
main.main()
	/etc/ente/cmd/museum/main.go:115 +0x3ac
INFO[0000]main.go:95 main Booting up local server with commit #b0674cd5f37cebda0f446db7ce1e87a3e6d8b9fc 
INFO[0000]main.go:791 setupDatabase Setting up db                                
INFO[0000]main.go:798 setupDatabase Connected to DB                              
INFO[0000]main.go:803 setupDatabase Pinged DB                                    
INFO[0000]main.go:812 setupDatabase Loaded migration scripts                     
INFO[0000]main.go:821 setupDatabase Database was configured successfully.        
INFO[0000]lock.go:65 ReleaseHostLock Released 0 locks held by c1ee1270eec7        
INFO[0000]s3config.go:114 initialize Hot storage: b2-eu-cen (secondary: wasabi-eu-central-2-v3) 
ERRO[0000]offer.go:49 NewOfferController Error reading config fileopen : no such file or directory 
ERRO[0000]offer.go:53 NewOfferController Error un-marshalling JSONunexpected end of JSON input 
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory 
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory 
INFO[0000]replication3.go:96 StartReplication replication.worker-url was not defined, files will downloaded directly during replication 
INFO[0000]replication3.go:146 createTemporaryStorage Temporary storage for replication v3 is: /replication/data 
INFO[0000]replicate.go:22 StartReplication replication.worker-url was not defined, file data will downloaded directly during replication 
INFO[0000]replication3.go:120 startWorkers Starting 6 workers for replication v3        
INFO[0000]object_cleanup.go:107 StartRemovingUnreportedObjects Starting 1 workers to remove-unreported-objects 
INFO[0000]delete.go:22 startDeleteWorkers Starting 1 delete workers for fileData       
INFO[0000]object_cleanup.go:132 removeUnreportedObjects Removing unreported objects                   task=remove-unreported-objects
INFO[0000]discord.go:56 sendMessage Skipping sending Discord message: c1ee1270eec7 has taken off 🚀 
INFO[0000]main.go:734 main We have lift-off.                            
INFO[0000]replicate.go:36 startWorkers Starting 6 workers for replication v3        
INFO[0000]repository.go:79 DeleteOldKeys Deleted old keys less than 1731834524645591 old 
INFO[0000]object_cleanup.go:152 removeUnreportedObjects Removed 0 objects                             task=remove-unreported-objects
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe74d2]

goroutine 74 [running]:
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).downloadObject(0xc0005d7ef0, {0x16b4fd0, 0xc000656f50}, {0xc000f14a20, 0x2a}, {0xc000f408a0, 0xc})
	/etc/ente/pkg/controller/filedata/s3.go:61 +0x1f2
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicateRowData(0xc0005d7ef0, {0x16b4fd0, 0xc000656f50}, {0x996c45, 0x59d82cb1ab006, {0xc000f40898, 0x6}, 0x2507, {0xc000f408a0, 0xc}, ...})
	/etc/ente/pkg/controller/filedata/replicate.go:97 +0x238
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).tryReplicate(0xc0005d7ef0)
	/etc/ente/pkg/controller/filedata/replicate.go:70 +0x230
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicate(0x0?, 0x0)
	/etc/ente/pkg/controller/filedata/replicate.go:50 +0x25
created by github.com/ente-io/museum/pkg/controller/filedata.(*Controller).startWorkers in goroutine 71
	/etc/ente/pkg/controller/filedata/replicate.go:39 +0x77

Answered by fredix

Feb 4, 2025

Hi all !
I removed all the mldata from my S3 bucket and from postgresql. And now the replication works !

for S3 bucket : aws s3 rm s3://fredix-ente/XXXXXX/file-data/ --recursive
for postgresql : delete from file_data where data_type='mldata';

View full answer

mngshm · 2024-11-17T13:22:16Z

mngshm
Nov 17, 2024
Collaborator

I see a :5432 connection error. It seems that the port is occupied, you should free up the port by killing whatever is occupying it.

Out of curiosity, Why is both 2nd and 3rd S3 endpoint pointing towards :3900 ? Is garage running inside docker container ?

4 replies

fredix Nov 17, 2024
Author

You can forget the connection error on 5432 I fixed it.
For the S3 endpoint I do not use docker, garage run on the hosts

BTW I edited my post because I use now use_path_style_urls: true

Sandeepbreddy Dec 4, 2024

@fredix how did you resolve the connection error ? am using the docker image of Museum, there is no replication setting up in my NAS. I also added support of IPv6 in compose file still seeing the same error with the museum

fredix Dec 4, 2024
Author

@fredix how did you resolve the connection error ? am using the docker image of Museum, there is no replication setting up in my NAS. I also added support of IPv6 in compose file still seeing the same error with the museum

I overwrite the entrypoint of the docker image, I use Nomad but it should be close with a docker-compose:

      config {
        image = "ghcr.io/ente-io/server"
        command = "/bin/sh"
        args = [
          "-c",
          "while ! nc -z 127.0.0.1 5432 </dev/null; do echo 'waiting for postgresql...'; sleep 5; done; /museum",
        ]

Sandeepbreddy Dec 8, 2024

Tried it but still throws the same error. panic:

dial tcp [::1]:5432: connect: connection refused

Am going through the code, normally Ping() is expecting host, am passing the docker hostname Postgres,

Am missing something in the credentials file ?

ua741 · 2024-12-05T04:22:59Z

ua741
Dec 5, 2024
Maintainer

Are you using the docker image or running latest main? There was a bug that was causing crash, but we have already fixed it in the main branch. IIRC, we didn't publish the bad image on docker registry.

5 replies

fredix Dec 5, 2024
Author

I'm using your docker image : "ghcr.io/ente-io/server"
Ok So I will build my own image from the main repo and check replication.

fredix Dec 5, 2024
Author

Hi, I built my on image from your main repo after a fresh git pull (https://hub.docker.com/r/fredix/ente) but it change nothing

INFO[0000]main.go:95 main Booting up local server with commit #        
INFO[0000]main.go:805 setupDatabase Setting up db                                
INFO[0000]main.go:812 setupDatabase Connected to DB                              
INFO[0000]main.go:817 setupDatabase Pinged DB                                    
INFO[0000]main.go:826 setupDatabase Loaded migration scripts                     
INFO[0000]main.go:835 setupDatabase Database was configured successfully.        
INFO[0001]lock.go:65 ReleaseHostLock Released 0 locks held by 504ae06381a0        
ERRO[0001]offer.go:49 NewOfferController Error reading config fileopen : no such file or directory 
ERRO[0001]offer.go:53 NewOfferController Error un-marshalling JSONunexpected end of JSON input 
ERRO[0001]billing.go:91 parsePricingFile Error reading file : open : no such file or directory 
ERRO[0001]billing.go:91 parsePricingFile Error reading file : open : no such file or directory 
INFO[0001]replication3.go:97 StartReplication replication.worker-url was not defined, files will downloaded directly during replication 
INFO[0001]replication3.go:147 createTemporaryStorage Temporary storage for replication v3 is: /replication/data 
INFO[0001]replicate.go:22 StartReplication replication.worker-url was not defined, file data will downloaded directly during replication 
INFO[0001]object_cleanup.go:107 StartRemovingUnreportedObjects Starting 1 workers to remove-unreported-objects 
INFO[0001]discord.go:56 sendMessage Skipping sending Discord message: 504ae06381a0 has taken off 🚀 
INFO[0001]main.go:748 main We have lift-off.                            
INFO[0001]replication3.go:121 startWorkers Starting 6 workers for replication v3        
INFO[0001]replicate.go:36 startWorkers Starting 6 workers for replication v3        
INFO[0001]delete.go:22 startDeleteWorkers Starting 1 delete workers for fileData       
INFO[0001]object_cleanup.go:132 removeUnreportedObjects Removing unreported objects                   task=remove-unreported-objects
INFO[0001]repository.go:79 DeleteOldKeys Deleted old keys less than 1733423467845158 old 
INFO[0001]object_cleanup.go:152 removeUnreportedObjects Removed 0 objects                             task=remove-unreported-objects
INFO[0002]replication3.go:268 tryReplicate Replication attempt start                     object_key=1580559962386438/f95f6e13-6650-4cf2-8a9c-7f08f7a3d61f task=replication
INFO[0002]replication3.go:268 tryReplicate Replication attempt start                     object_key=1580559962386438/80158b47-1ea2-4db6-a7c9-f99ae3ac16ac task=replication
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe95b2]

goroutine 47 [running]:
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).downloadObject(0xc000591180, {0x16b91d0, 0xc00117e150}, {0xc00053b230, 0x2a}, {0xc000e28e10, 0xc})
	/etc/ente/pkg/controller/filedata/s3.go:61 +0x1f2
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicateRowData(0xc000591180, {0x16b91d0, 0xc00117e150}, {0x996724, 0x59d82cb1ab006, {0xc000e28e00, 0x6}, 0x250b, {0xc000e28e10, 0xc}, ...})
	/etc/ente/pkg/controller/filedata/replicate.go:97 +0x238
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).tryReplicate(0xc000591180)
	/etc/ente/pkg/controller/filedata/replicate.go:70 +0x230
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicate(0x51abc5?, 0x1)
	/etc/ente/pkg/controller/filedata/replicate.go:50 +0x25
created by github.com/ente-io/museum/pkg/controller/filedata.(*Controller).startWorkers in goroutine 74
	/etc/ente/pkg/controller/filedata/replicate.go:39 +0x77

ua741 Dec 6, 2024
Maintainer

It seems like you had configured something to replicate the mldata, and later on removed config? These is trying to replicate data related to ml embedding, not the actual file objects. Can you confirm if you remember making any change related to s3.derived-storage or s3.file-data-config?

fredix Dec 6, 2024
Author

I don't know what is 3.derived-storage nor s3.file-data-config

fredix Dec 16, 2024
Author

Hi,
If I empty the local S3 buckets and resynchronize them with rclone from the Scaleway bucket (primary), will this work?

fredix · 2025-02-04T09:43:24Z

fredix
Feb 4, 2025
Author

Hi all !
I removed all the mldata from my S3 bucket and from postgresql. And now the replication works !

for S3 bucket : aws s3 rm s3://fredix-ente/XXXXXX/file-data/ --recursive
for postgresql : delete from file_data where data_type='mldata';

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

museum crash when replication enabled #4063

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 9 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

museum crash when replication enabled #4063

fredix Nov 17, 2024

Replies: 3 comments · 9 replies

mngshm Nov 17, 2024 Collaborator

fredix Nov 17, 2024 Author

Sandeepbreddy Dec 4, 2024

fredix Dec 4, 2024 Author

Sandeepbreddy Dec 8, 2024

ua741 Dec 5, 2024 Maintainer

fredix Dec 5, 2024 Author

fredix Dec 5, 2024 Author

ua741 Dec 6, 2024 Maintainer

fredix Dec 6, 2024 Author

fredix Dec 16, 2024 Author

fredix Feb 4, 2025 Author

fredix
Nov 17, 2024

Replies: 3 comments 9 replies

mngshm
Nov 17, 2024
Collaborator

fredix Nov 17, 2024
Author

fredix Dec 4, 2024
Author

ua741
Dec 5, 2024
Maintainer

fredix Dec 5, 2024
Author

fredix Dec 5, 2024
Author

ua741 Dec 6, 2024
Maintainer

fredix Dec 6, 2024
Author

fredix Dec 16, 2024
Author

fredix
Feb 4, 2025
Author