Importing large amount of data in a pre-existing database

[aladdin1]

Hello,

How can I import a large amount of new data in a pre-existing database. My table are already partially populated, I do not want duplicate. I try using the "Import" feature but it does not work with already 25.000 rows (yes my text file is well formated, the same data are successfully imported when the table is empty), I try using the "Find Duplicate" feature but it does not work either for the same reason. I cannot use Access to pre-clean my data because the table is too large. What is the solution to my problem?

[FSGDAG]

Have you tried breaking down the data into multiple smaller files??? When I imported my 10,000 links, I broke them down into 10 files...

Hopefully, thats what you mean

[aladdin1]

I try to import as little as 50 rows at a time without success. With 14.000 row to go even if it was working (50 rows) it would not be practical.

[Bruceper]

Do you have external access to your database? If so you can use a tool like MySQL front to import the file, and it will do it for you without timing out.

[aladdin1]

I try to import my new data using "phpmyadmin" but I do get an error message every time I hit a duplicate.

[inspireme]

phpmyadmin shouldnt complain about duplicates unless you have the link_id column popualted with numbers.

What i would do is export your current databas to an excel spreadsheet from phpmyadmin. Get your other items you want to add also in a spreadsheet.then do some filtering in excel to get rid of duplicates and tidy it up. Export to a comma separated csv, empty current idx_link (aafter backup!) then import the csv file again

[aladdin1]

Thanks for your suggestion, My database is too big, Excel cannot handle a 100 Mb spreadsheet.
I did find an alternate solution, I did load my new data in idx_link_temp then I validate my new link. Since I made my "title" field unique (I work on an article directory), MySQL reject duplicates title after they are validate. Not very efficient but it is the best solution I did find, it take about 30 minutes for 1000 links.
You can see my Sure tips Articles directory at Sure Tips Article Directory, tell me what you think.

[inspireme]

excel? hmm, i open 1gb+ on mine... i know open office really struggles though. glad you got it working - good site :0)

[aladdin1]

excel? hmm, i open 1gb+ on mine... i know open office really struggles though. glad you got it working - good site :0)

I should confess that I never try to open my excel file with Open Office, I usually use Microsoft Office but I shall try, thanks for the tip. Open Office access my be able to help. I shall let you know.