Scraper stops every few entries

MariMa · 4 October 2022 11:03

I bought new Odroid N2+ to replace my old, and no longer supported C2 and I decided to rebuild the database to make a clean fresh start with the new hardware. As database I have chosen the official mariadb docker image in version 10.5.9, which already ran well before. The database user has been set up as follows

CREATE SCHEMA kodi;
CREATE USER ‘kodi’ IDENTIFIED BY ‘password’;
GRANT ALL ON kodi%.* TO ‘kodi’;
FLUSH PRIVILEGES;

The following has been configured in advancedsettings.xml:

        <videodatabase>
                <type>mysql</type>
                <host>192.168.0.9</host>
                <port>3306</port>
                <user>kodi</user>
                <pass>password</pass>
                <name>kodi_video</name>
        </videodatabase>
        <musicdatabase>
                <type>mysql</type>
                <host>192.168.0.9</host>
                <port>3306</port>
                <user>kodi</user>
                <pass>password</pass>
                <name>kodi_music</name>
        </musicdatabase>

The database is also set up after the first start of Kodi so I think everything is configured correctly.

Since I have created a file set (poster, fanart and nfo) for each video with MediaElch version 2.8.16, I select in the menu ‘Set content’ as information provider ‘Local information only’.

Now the scraper starts to fill the database, but not for long and the scraper stops. This happens every 1 to 10 entries. A regularity is not to be recognized. For example, if the scraper stops at movie 5, it could also stop at movie 3 or 8 after the next reinstallation. If the scraper does not run any more, the symbol with the percentage is still displayed.

I can recreate this behavior with:

Odroid C2
CoreELEC-Amlogic.arm-9.2.8-Odroid_C2.img

Odroid C4
CoreELEC-Amlogic-ng.arm-19.5-Matrix_rc2-Odroid_C4.img

Odroid N2+
CoreELEC-Amlogic-ng.arm-19.5-Matrix_nightly_20220916-Odroid_N2.img
CoreELEC-Amlogic-ng.arm-19.5-Matrix_nightly_20220929-Odroid_N2.img
CoreELEC-Amlogic-ng.arm-19.5-Matrix_rc2-Odroid_N2.img

It was always detected immediately after a reinstall without add-ons installed.

The same problem exists or also when I select ‘The Movie Database’ as scraper.

I have also installed MariaDB version 10.5 directly on my Synology NAS → Same problem.

There is no ERROR in the log files at this point, it just stops. I will gladly provide the kodi.log file, if desired.

Maybe someone here has experienced something similar and knows a solution.

matt77303 · 4 October 2022 14:58

i gave up with this setup and installed jellyfin on my synology NAS instead.

Never looked back.

Matt

MariMa · 4 October 2022 15:06

Hello Matt,
I understand you very well. I am also on the verge of giving up. I bought 4 N2+, full of joy, everything would run faster now and now nothing runs at all.
For over a week I’m trying to scrape all the movies again. As written, two scrape, reboot, one scrape, reboot, maybe times five scrape, reboot…

I’m afraid I don’t understand why everyone here is raving about the N2+. It just doesn’t work with MySQL database.
But maybe I’m just doing something wrong, just what?

Re-installation
basic configuration
scrape
scraper stops → reboot

I’ll have a look at Jellyfin, thanks for the tip.

Vasco · 4 October 2022 15:16

My setup is multiple devices with a Mariadb database on my HC4 server. Never had any major problem.

MariMa · 4 October 2022 15:22

Hi Vasco,
I believe you and also my Odroid C2 were running without any issues over years. Btw with the same MariaDB, that’s why I think the MariaDB isn’t the reason of the problems. The problems started with the change to N2+. But as I wrote I have the same issue with a Odroid C4.
I believe I did something wrong, but what?

Vasco · 4 October 2022 15:27

Not easy to say, but as I said I used different devices where N2+ and C4 are included. Just followed the Kodi wiki to “assemble” it with probably some tweaks.

MariMa · 4 October 2022 15:36

If at least the kodi.log would show something.
I have in the advancedsettings.xml
<loglevel>2</loglevel>
but still nothing useful appears in the logs.

MariMa · 5 October 2022 09:12

I have tested further and can now rule out the following root causes:

Hardware:

3 Odroid N2+ and 1 Odroid C4 have the same problem.
=> unlikely that all are defective

Video files:

Sometimes a video file is scanned clean and sometimes not
All can be played without problems
=> Unlikely that all are defective

Storage media:

Three SanDisk Ultra 16GB microSDHC Class 10, U1, A1 tested.
eMMC 16 GB tested
=> Unlikely that all are defective

Databases:

Existing MariaDB 10.5 in Docker container tested.
New installation MariaDB 10.5 on Synology NAS tested
New installation MariaDB 10.5 on Odroid C4 with DietPi OS tested
=> Unlikely that all are defective

I also deleted all posters, fanart and NFO files and directly scanned with ‘The Movie Database’ Scraper
=> Got seven videos scanned until scraper stops

Kogi Log:
2022-10-05 10:49:22.550 T:3703 DEBUG : Thread JobWorker 3804213888 terminating (autodelete)
2022-10-05 10:49:22.550 T:3724 DEBUG : Thread JobWorker 3643867776 terminating (autodelete)
2022-10-05 10:49:45.991 T:3656 DEBUG : CheckIdle - Closing session to https://image.tmdb.org (easy=0xe58a5678, multi=0xe58cf218)
2022-10-05 10:49:49.091 T:3656 DEBUG : CheckIdle - Closing session to https://api.themoviedb.org (easy=0x2699698, multi=0x26a6e00)

I really don’t know what to do anymore. Maybe someone can do something with the log entries. Gladly I provide also the full logs

Vasco · 5 October 2022 09:41

Which scrapper exactly are you using?

MariMa · 5 October 2022 10:09

Hello Vasco,
thanks for your reply
I am using the default scraper ‘The. Movie Database / TMDB Moviescraper’ but get the same problems when using the second default scraper ‘The Movie Database Python / TMDB Moviescraper’ and it is running…

MariMa · 5 October 2022 10:11

At this moment I tested The Movie Database Python scraper again and it stopped after ~60 videos. My personal record

Vasco · 5 October 2022 10:38

Use the python one as the other is deprecated and used only on Leia. And provide logs for that one, maybe something can be found.

MariMa · 5 October 2022 10:59

Super, I will do so.
I will do a new kodi installation now without a database. Only a created db user, then I run throw the setup wizard without any changes, advancedsettings.xml and sources.xml will be transferred via scp and then I reboot. On restart the database will be created. After that I start scraping using the Movie Database Python scraper until it stops. How can I provide the log? Attach to this topic?

Or use ‘Submit Log’ setting?

MariMa · 6 October 2022 16:45

I have now tested pretty much all possibilities, even with another scraper (Universal Movie Scraper) I have the same problem.
The one thing that was always the same in my tests is the Coreelec image, downloaded multiple times, so I think that’s where the error lies.

With NFO files in the video folder I get about 1 to 10 videos far, without NFO files in the folder ca.40 at the beginning and then also always only a few videos further. In between always reboots.
I think no one has rebooted Coreelec more often than me in the past few days. Worldwide.

Too bad, but I think I will then probably have to plan further without Coreelec.

Is there anyone here interested in my hardware?

3 ODROID N2+
4 ODROID C2
1 ODROID C4

Vasco · 6 October 2022 17:58

You forgot another thing that’s always the same, the user. You have hundreds or thousands of CE users doing exactly what you’re trying to do and achieving it without any problem. Have you tried to scrape only part of your library? Cutting it in half and try both to see if it’s some of your movies that is messing with the system? If you just repeat the procedure everytime you’ll always get the same result. Have you tried to connect a HDD with USB and try to scrape locally? There are many things you can try but ok, just blame CE, we can’t please everyone.

MariMa · 7 October 2022 05:09

Dear Vasco,
unfortunately you are right. The error lies somewhere on my side and I have also mentioned several times. I don’t want to sugarcoat it, and I certainly don’t want to blame Coreelec. I’m sorry if it came across that way. I have been using Corelec for many years and have always been very satisfied with it.
I know, never change a running system, but the used C2 had to be changed after years to use the current version of Corelec.

I have of course already carried out the tests you mentioned. Library splitted and also with the existing nfo files in the locale SQLite database written. This works WITHOUT problems, so I could conclude that the videos and also the nfo files are okay.
But surely you agree with me that using a local SQLite database with multiple Coreelec installations in the house is not the optimal way, but just using a central database.

And only when I was stuck for days, I registered here in the forum, hoping to get help. So I am still grateful for any tips and help.

MariMa · 7 October 2022 11:08

I found the root cause, It is working now as expected
@matt77303: Thanks for your recommendation to have a look at JellyFin. Nice backup solution but the combination Coreelec + MariaDB is in my point of view a better solution

Thanks also to the Coreelec team for your help

Vasco · 7 October 2022 11:11

Wonderful news, can you share the solution/problem if someone faces a similar issue?

MariMa · 7 October 2022 11:13

Yes, for sure. It was the following entry in my advancedsettings.xml:

        <pathsubstitution>
                <substitute>
                        <from>special://masterprofile/Thumbnails</from>
                        <to>nfs://192.168.0.254/volume2/kodi/kodi_thumbs/</to>
                </substitute>
        </pathsubstitution>

Vasco · 7 October 2022 11:16

Ah, you use shared thumbs. Good catch.