Django loaddata error - Исправление ошибок и поиск оптимальных решений проблем

I generated a fixture:

python manage.py dumpdata --all > ./mydump.json

I emptied all my databases using:

python manage.py sqlflush | psql mydatabase -U mydbuser

But when i try to use loaddata:

python manage.py loaddata ./mydump.json

I’m recieving this error:

IntegrityError: Could not load tastypie.ApiKey(pk=1): duplicate key 
value violates unique constraint "tastypie_apikey_user_id_key" 
DETAIL:  Key (user_id)=(2) already exists.

I’m having this problem on production and i’m out of ideas. Someone had a similar problem?

asked Feb 9, 2014 at 2:41

Adrian LopezAdrian Lopez

2,4615 gold badges30 silver badges48 bronze badges

Run loaddata with all @recievers commented out because they will be fired when loaddata loads your data. If @recievers create other objects as a sideeffect it will cause collisions.

answered Feb 28, 2020 at 20:50

Jan KaiferJan Kaifer

6524 silver badges18 bronze badges

First:
I believe your unix pipe is incorrectly written.

# 1: Dump your json
$ python manage.py dumpdata --all > ./mydump.json

# 2: dump your schema
$ python manage.py sqlflush > schema.sql

# 3: launch psql
# this is how I launch psql ( seems to be more portable between rhel/ubuntu )
# you might use a bit different technique, and that is ok.

Edited: (very important)
Make sure you do not have any active django connections running on your server. Then:

$ sudo -u myuser psql mydatabase

# 4: read in schema
mydatabase=# i schema.sql
mydatabase=# ctrl-d

# 5: load back in your fixture. 
$ python manage.py loaddata ./mydump.json

Second:
If your pipe is ok.. and it might be. Depending on your schema/data you may need to use natural-keys.

# 1: Dump your json using ( -n ) natural keys.
$ python manage.py dumpdata -n --all > ./mydump.json

# followed by steps 2-5 above.

answered Feb 10, 2014 at 6:51

Jeff SheffieldJeff Sheffield

5,5983 gold badges25 silver badges31 bronze badges

Jeff Sheffield’s solution is correct, but now I find that a solution like django-dbbackup is by far the most generic and simplier way to do it with any database.

python manage.py dbbackup

answered Apr 4, 2014 at 4:12

Adrian LopezAdrian Lopez

2,4615 gold badges30 silver badges48 bronze badges

Источник

I created a «fixtures» folder in the app directory and put data1.json in there.

This is what is in the file:

[{"firm_url": "http://www.graychase.com/kadam", "firm_name": "Gray & Chase", "first": " Karin ", "last": "Adam", "school": "Ernst Moritz Arndt University Greifswald",  "year_graduated": " 2004"} ]

In the command line I cd to the app directory and

django-admin.py loaddata data1.json

but I get this error

Installing json fixture 'data1' from
'C:UsersADocumentsProjectsDjangosw2wkw2fixtures'.
Problem installing fixture
'C:UsersADocumentsProjectsDjangosw2wkw2fixturesdata1.json': Traceback (most recent call last):
File "C:Python26Libsite-packagesdjangocoremanagementcommandsloaddata.py", line 150, in handle for obj in objects:
File "C:Python26libsite-packagesdjangocoreserializersjson.py", line 41, in Deserializer for obj in PythonDeserializer(simplejson.load(stream)):
File "C:Python26libsite-packagesdjangocoreserializerspython.py", line 76, in Deserializer
Model = _get_model(d["model"])
KeyError: 'model'

What am I doing wrong?

Edit:

I fixed the json format:

[
    {
        "pk": 1, 
        "model": "wkw2.Lawyer", 
        "fields": {
            "school": "The George Washington University Law School", 
            "last": "Babas", 
            "firm_url": "http://www.graychase.com/babbas", 
            "year_graduated": "2005", 
            "firm_name": "Gray & Chase", 
            "first": "Amr A"
        }
    }
]

But now I get ValidationError: This value must be an integer. Is there a way to find out from the line numbers what causes the error? Only «pk» is an integer.

Problem installing fixture 'C:~sw2wkw2fixturescsvtest1.csv.json': Traceback (most recent call last):
File "C:Python26Libsite-packagesdjangocoremanagementcommandsloaddata.py", line 150, in handle for obj in objects:
File "C:Python26libsite-packagesdjangocoreserializersjson.py", line 41, in  Deserializer for obj in PythonDeserializer(simplejson.load(stream)):
File "C:Python26libsite-packagesdjangocoreserializerspython.py", line 95, in Deserializer data[field.attname] =  field.rel.to._meta.get_field(field.rel.field_name).to_python(field_value)
File "C:Python26libsite-packagesdjangodbmodelsfields__init__.py", line 356, in to_python_("This value must be an integer."))

ValidationError: This value must be an integer.

Источник

When migrating a Django project it’s often useful to work with the built-in
dumpdata and loaddata management commands.
However, this can easily lead to errors like django.db.utils.IntegrityError:
Problem installing fixture Could not load foo.Bar(pk=ba): (1062,
«Duplicate entry for key ‘app_label'»).
Such errors indictate that a similar object already exists in the database,
and that you can’t insert the duplicate from your fixture.

There are basically two approaches to dump data in a way that solves this problem:

Specify exactly which models to dump
Exclude the conflicting models from the dump

For the first approach you would use a command like:

./manage.py dumpdata --natural --indent=2 app1 app2 app3 auth.User > dump.json

I include the auth data as well in this example, because it’s what I usually
do.
You want to include all third party apps that create their own database tables
and store user generated content.
You want to avoid apps that auto-populate the database, like thumbnail
generators, search indexers, etc.

The second approach is to exclude models. You can do this like:

./manage.py dumpdata --natural --indent=2 --exclude=contenttypes --exclude=sessions.Session --exclude=south.Migrationhistory > dump.json

If you googled the error
django.db.utils.IntegrityError: Problem installing fixture Could not load contenttypes.ContentType(pk=X): (1062, «Duplicate entry for key ‘app_label'»).
you probably want to exclude contenttypes.
The example above also exlcudes live sessions and south migrationhistory.

Loading the fixtures

To load the fixtures you need to prepare the database on your new host.
First:

./manage.py syncdb --noinput --no-initial-data

I use the —no-initial-data option by default. You might want to remove
the option if some of your apps use fixtures, and you didn’t include those apps in your
data dump.

To finish, apply the migrations and load the data:

./manage.py migrate
./manage.py loaddata dump.json

If you still get errors, read them, and think about which data to
include/exclude in the dump. Remeber to delete all objects in the database though before you try a new import. The sqlclear management command can help with that. Or you can simply DROP the database and CREATE it again.

Источник

#json #django #loaddata

Вопрос:

здравствуйте, я столкнулся с проблемой, из-за которой FFMPEG не установлен, я работаю как на локальном, так и на производстве, оба работают на локальном python manage.py категории модерации loaddata.json отлично работает, когда на производстве выполняются одни и те же команды, у меня возникла проблема

ОШИБКИ: FFmpegBackend’>: (video_conversion.E001) Двоичный файл ffmpeg не найден: ПОДСКАЗКА: Пожалуйста, установите ffmpeg. FFmpegBackend’>: (video_conversion.E001) Двоичный файл ffmpeg не найден: ПОДСКАЗКА: Пожалуйста, установите ffmpeg.

во время выполнения команды

 docker-compose -f docker-compose-full.yml run --rm webserver python3 manage.py loaddata moderation_categories.json

у меня есть обратная трассировка ошибок:

  Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/json.py", line 68, in Deserializer
    objects = json.loads(stream_or_string)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/modeltranslation/management/commands/loaddata.py", line 61, in handle
    return super(Command, self).handle(*fixture_labels, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/json.py", line 73, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/opt/siuu-api/moderation_categories.json':

пожалуйста, дайте мне какое-нибудь решение здесь
, используя python 3.8.5

Источник

When you’re using MySQL with InnoDB for unit-testing, you may stumbled into this problems when try loading fixtures in test case.

1	IntegrityError: (1452, ‘Cannot add or update a child row: a foreign key constraint fails (`DATABASE`.`TABLE_NAME`, CONSTRAINT `COLUMN_id_refs_id_ea62e552` FOREIGN KEY (`room_id`) REFERENCES `ANOTHER_TABLE` (`id`))’)

You may solve this problem with this quickfix solution:

1. Change your database ENGINE into MyISAM when running unit-testing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

options = «INNODB»
import sys
if ‘tests’ in sys.argv:
options = «MYISAM»

DATABASES = {
‘default’: {
‘ENGINE’: ‘django.db.backends.mysql’,
‘NAME’: ‘tripvillas_dev’,
‘USER’: ‘root’,
‘PASSWORD’: ‘trip’,
‘HOST’: »,
‘PORT’: »,
‘OPTIONS’: {‘init_command’: ‘SET storage_engine=’ + options},
},
}

2. Using SQLITE when running unit testing

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import sys
if ‘test’ in sys.argv:
import os
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__))

DATABASES = {
‘default’: {
‘ENGINE’: ‘django.db.backends.sqlite3’,
# ‘NAME’: os.path.join(PROJECT_PATH, ‘mytesting.db’),
‘PASSWORD’: »,
‘HOST’: »,
‘PORT’: »,
},
}

3. SET_FOREIGN_CHECKS=0
Yes, when you running “SET_FOREIGN_CHECKS=0;” in your MySQL, it will make INNODB ignoring foreign checks when storing new record.

1	‘OPTIONS’: {‘init_command’: ‘SET set_foreign_checks=0;’}

That’s all just quickfix but not “REALLY” see what the caused. Well, actually there no problem with foreign key records in Django 1.3.

This problem may occured because incorrect position of data in fixtures. You should know that django loaddata will parse and executing your fixtures LINE BY LINE.

For instance, if you have models like :

1
2
3
4
5

class Insurance(models.Model):
company=models.ForeignKey(Company)

class Company(models.Model):
name=models.Charfield()

And you have Has many to many relation models. You can export them into FIXTURES and running without problem.
But yes, the problem will came out if you have wrong ordering data, because when you load fixtures, django will read all data on each line and executed them.

This is the exported data sample (in YAML):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

— fields:
company: [6d54eaa83bf34b6d9b1b34ecef069e60, c22d619375d14941a40b0749f8a9c391]
model: finance.insurance
pk: ‘10017060’

— fields:
name: First Company
model: finance.company
pk: c22d619375d14941a40b0749f8a9c391

— fields:
name: Second Company
model: finance.company
pk: 6d54eaa83bf34b6d9b1b34ecef069e60

When you execute or load fixtures with this way, then it will lead you to “IntegrityError: (1452, ‘Cannot add or update a child row: a foreign key constraint fails”. This is happen because once Insurance created, it will try to find Company Foreign keys.

To solve this problems, re-arrage the position:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

— fields:
name: First Company
model: finance.company
pk: c22d619375d14941a40b0749f8a9c391

— fields:
name: Second Company
model: finance.company
pk: 6d54eaa83bf34b6d9b1b34ecef069e60

— fields:
company: [6d54eaa83bf34b6d9b1b34ecef069e60, c22d619375d14941a40b0749f8a9c391]
model: finance.insurance
pk: ‘10017060’

This this fixtures will be loaded successfully, because when Insurance created, it will get the ForeignKey. Remember, you doesn’t need to export the has many table relation at this cases.

Using “natural keys” (param –natural) to represent any foreign key and many-to-many relationship with a model that provideds a natural.

At bottom line here, Django 1.3 shouldn’t have Foreign Key problems when loading fixtures 🙂

Источник