Input contains nan infinity or a value too large for dtype float64 как исправить

When using a dataset for analysis, you must check your data to ensure it only contains finite numbers and no NaN values (Not a Number). If you try to pass a dataset that contains NaN or infinity values to a function for analysis, you will raise the error: ValueError: input contains nan, infinity or a value too large for dtype(‘float64’).

To solve this error, you can check your data set for NaN values using numpy.isnan() and infinite values using numpy.isfinite(). You can replace NaN values using nan_to_num() if your data is in a numpy array or SciKit-Learn’s SimpleImputer.

This tutorial will go through the error in detail and how to solve it with the help of code examples.

Python ValueError: input contains nan, infinity or a value too large for dtype(‘float64’)

What is a ValueError?

In Python, a value is the information stored within a particular object. You will encounter a ValueError in Python when you use a built-in operation or function that receives an argument with the right type but an inappropriate value.

What is a NaN in Python?

In Python, a NaN stands for Not a Number and represents undefined entries and missing values in a dataset.

What is inf in Python?

Infinity in Python is a number that is greater than every other numeric value and can either be positive or negative. All arithmetic operations performed on an infinite value will produce an infinite number. Infinity is a float value; there is no way to represent infinity as an integer. We can use float() to represent infinity as follows:

pos_inf=float('inf')

neg_inf=-float('inf')

print('Positive infinity: ', pos_inf)

print('Negative infinity: ', neg_inf)

Positive infinity:  inf
Negative infinity:  -inf

We can also use the math, decimal, sympy, and numpy modules to represent infinity in Python.

Let’s look at some examples where we want to clean our data of NaN and infinity values.

Example #1: Dataset with NaN Values

In this example, we will generate a dataset consisting of random numbers and then randomly populate the dataset with NaN values. We will try to cluster the values in the dataset using the AffinityPropagation in the Scikit-Learn library.

Note: The use of the AffinityPropagation to cluster on random data is just an example to demonstrate the source of the error. The function you are trying to use may be completely different to AffinityPropagation, but the data preprocessing described in this tutorial will still apply.

The data generation looks as follows:

# Import numpy and AffinityPropagation

import numpy as np

from sklearn.cluster import AffinityPropagation

# Number of NaN values to put into data

n = 4

data = np.random.randn(20)

# Get random indices in the data

index_nan = np.random.choice(data.size, n, replace=False)

# Replace data with NaN

data.ravel()[index_nan]=np.nan

print(data)

Let’s look at the data:

[-0.0063374  -0.974195    0.94467842  0.38736788  0.84908087         nan
  1.00582645         nan  1.87585201 -0.98264992 -1.64822932  1.24843544
  0.88220504 -1.4204208   0.53238027         nan  0.83446561         nan
 -0.04655628 -1.09054183]

The data consists of twenty random values, four of which are NaN, and the rest are numerical values. Let’s try to fit the data using the AffinityPropagation() class.

af= AffinityPropagation(random_state=5).fit([data])

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

We raise the error because the AffinityPropagation.fit() cannot handle NaN, infinity or extremely large values. Our data contains NaN values, and we need to preprocess the data to replace them with suitable values.

Solution #1: using nan_to_num()

To check if a dataset contains NaN values, we can use the isnan() function from NumPy. If we pair this function with any(), we will check if there are any instances of NaN. We can replace the NaN values using the nan_to_num() method. Let’s look at the code and the clean data:

print(np.any(np.isnan(data)))

data = np.nan_to_num(data)

print(data)

True
[-0.0063374  -0.974195    0.94467842  0.38736788  0.84908087  0.
  1.00582645  0.          1.87585201 -0.98264992 -1.64822932  1.24843544
  0.88220504 -1.4204208   0.53238027  0.          0.83446561  0.
 -0.04655628 -1.09054183]

The np.any() part of the code returns True because our dataset contains at least one NaN value. The clean data has zeros in place of the NaN values. Let’s fit on the clean data:

af= AffinityPropagation(random_state=5).fit([data])

This code will execute without any errors.

Solution #2: using SimpleImputer

Scikit-Learn provides a class for imputation called SimpleImputer. We can use the SimpleImputer to replace NaN values. To replace NaN values in a one-dimensional dataset, we need to set the strategy parameter in the SimpleImputer to constant. First, we will generate the data:

import numpy as np

n = 4

data = np.random.randn(20)

index_nan = np.random.choice(data.size, n, replace=False)

data.ravel()[index_nan]=np.nan

print(data)

The data looks like this:

[ 1.4325319   0.61439789  0.3614522   1.38531346         nan  0.6900916
  0.50743745  0.48544145         nan         nan  0.17253557         nan
 -1.05027802  0.09648188  1.15971533  0.29005307  2.35040023  0.44103513
 -0.03235852 -0.78142219]

We can use the SimpleImputer class to fit and transform the data as follows:

from sklearn.impute import SimpleImputer

imp_mean = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)

imputer = imp_mean.fit([data])

data = imputer.transform([data])

print(data)

The clean data looks like this:

[[ 1.4325319   0.61439789  0.3614522   1.38531346  0.          0.6900916
   0.50743745  0.48544145  0.          0.          0.17253557  0.
  -1.05027802  0.09648188  1.15971533  0.29005307  2.35040023  0.44103513
  -0.03235852 -0.78142219]]

And we can pass the clean data to the AffinityPropagation clustering method as follows:

af= AffinityPropagation(random_state=5).fit(data)

We can also use the SimpleImputer class on multi-dimensional data to replace NaN values using the mean along each column. We have to set the imputation strategy to “mean”, and using the mean is only valid for numeric data. Let’s look at an example of a 3×3 nested list that contains NaN values:

from sklearn.impute import SimpleImputer

data = [[7, 2, np.nan], 
        [4, np.nan, 6], 
        [10, 5, 9]]

We can replace the NaN values as follows:

imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')

imp_mean.fit(data)

data = imp_mean.transform(data)

print(data)

[[ 7.   2.   7.5]
 [ 4.   3.5  6. ]
 [10.   5.   9. ]]

We replaced the np.nan values with the mean of the real numbers along the columns of the nested list. For example, in the third column, the real numbers are 6 and 9, so the mean is 7.5, which replaces the np.nan value in the third column.

We can use the other imputation strategies media and most_frequent.

Example #2: Dataset with NaN and inf Values

This example will generate a dataset consisting of random numbers and then randomly populate the dataset with NaN and infinity values. We will try to cluster the values in the dataset using the AffinityPropagation in the Scikit-Learn library. The data generation looks as follows:

import numpy as np

from sklearn.cluster import AffinityPropagation

n = 4

data = np.random.randn(20)

index_nan = np.random.choice(data.size, n, replace=False)

index_inf = np.random.choice(data.size, n, replace=False)

data.ravel()[index_nan]=np.nan

data.ravel()[index_inf]=np.inf

print(data)

[-0.76148741         inf  0.10339756         nan         inf -0.75013509
  1.2740893          nan -1.68682986         nan  0.57540185 -2.0435754
  0.99287213         inf  0.5838198          inf -0.62896815 -0.45368201
  0.49864775 -1.08881703]

The data consists of twenty random values, four of which are NaN, four are infinity, and the rest are numerical values. Let’s try to fit the data using the AffinityPropagation() class.

af= AffinityPropagation(random_state=5).fit([data])

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

We raise the error because the dataset contains NaN values and infinity values.

Solution #1: Using nan_to_num

To check if a dataset contains NaN values, we can use the isnan() function from NumPy. If we pair this function with any(), we will check if there are any instances of NaN.

To check if a dataset contains infinite values, we can use the isfinite() function from NumPy. If we pair this function with any(), we will check if there are any instances of infinity.

We can replace the NaN and infinity values using the nan_to_num() method. The method will set NaN values to zero and infinity values to a very large number. Let’s look at the code and the clean data:

print(np.any(np.isnan(data)))

print(np.all(np.isfinite(data)))

data = np.nan_to_num(data)

print(data)

True

False

[-7.61487414e-001  1.79769313e+308  1.03397556e-001  0.00000000e+000
  1.79769313e+308 -7.50135085e-001  1.27408930e+000  0.00000000e+000
 -1.68682986e+000  0.00000000e+000  5.75401847e-001 -2.04357540e+000
  9.92872128e-001  1.79769313e+308  5.83819800e-001  1.79769313e+308
 -6.28968155e-001 -4.53682014e-001  4.98647752e-001 -1.08881703e+000]

We replaced the NaN values with zeroes and the infinity values with 1.79769313e+308. We can fit on the clean data as follows:

af= AffinityPropagation(random_state=5).fit([data])

This code will execute without any errors. If we do not want to replace infinity with a very large number but with zero, we can convert the infinity values to NaN using:

data[data==np.inf] = np.nan

And then pass the data to the nan_to_num method, converting all the NaN values to zeroes.

Solution #2: Using fillna()

We can use Pandas to convert our dataset to a DataFrame and replace the NaN and infinity values using the Pandas fillna() method. First, let’s look at the data generation:

import numpy as np

import pandas as pd

from sklearn.cluster import AffinityPropagation

n = 4

data = np.random.randn(20)

index_nan = np.random.choice(data.size, n, replace=False)

index_inf = np.random.choice(data.size, n, replace=False)

data.ravel()[index_nan]=np.nan

data.ravel()[index_inf]=np.inf

print(data

[ 0.41339801         inf         nan  0.7854321   0.23319745         nan
  0.50342482         inf -0.82102161 -0.81934623  0.23176869 -0.61882322
  0.12434801 -0.21218049         inf -1.54067848         nan  1.78086445
         inf  0.4881174 ]

The data consists of twenty random values, four of which are NaN, four are infinity, and the rest are numerical values. We can convert the numpy array to a DataFrame as follows:

df = pd.DataFrame(data)

Once we have the DataFrame, we can use the replace method to replace the infinity values with NaN values. Then, we will call the fillna() method to replace all NaN values in the DataFrame.

df.replace([np.inf, -np.inf], np.nan, inplace=True)

df = df.fillna(0)

We can use the to_numpy() method to convert the DataFrame back to a numpy array as follows:

data = df.to_numpy()

print(data)

[[ 0.41339801]
 [ 0.        ]
 [ 0.        ]
 [ 0.7854321 ]
 [ 0.23319745]
 [ 0.        ]
 [ 0.50342482]
 [ 0.        ]
 [-0.82102161]
 [-0.81934623]
 [ 0.23176869]
 [-0.61882322]
 [ 0.12434801]
 [-0.21218049]
 [ 0.        ]
 [-1.54067848]
 [ 0.        ]
 [ 1.78086445]
 [ 0.        ]
 [ 0.4881174 ]]

We can now fit on the clean data using the AffinityPropagation class as follows:

af= AffinityPropagation(random_state=5).fit(data)

print(af.cluster_centers_)

The clustering algorithm gives us the following cluster centres:

[[ 0.        ]
 [ 0.50342482]
 [-0.81934623]
 [-1.54067848]
 [ 1.78086445]]

We can also use Pandas to drop columns with NaN values using the dropna() method. For further reading on using Pandas for data preprocessing, go to the article: Introduction to Pandas: A Complete Tutorial for Beginners.

Solution #3: using SimpleImputer

Let’s look at an example of using the SimpleImputer to replace NaN and infinity values. First, we will look at the data generation:

import numpy as np

n = 4

data = np.random.randn(20)

index_nan = np.random.choice(data.size, n, replace=False)

index_inf = np.random.choice(data.size, n, replace=False)

data.ravel()[index_nan]=np.nan

data.ravel()[index_inf]=np.inf

print(data)

[-0.5318616          nan  0.12842066         inf         inf         nan
  1.24679674  0.09636847  0.67969774  1.2029146          nan  0.60090616
 -0.46642723         nan  1.58596659  0.47893738  1.52861316         inf
 -1.36273437         inf]

The data consists of twenty random values, four of which are NaN, four are infinity, and the rest are numerical values. Let’s try to use the SimpleImputer to clean our data:

from sklearn.impute import SimpleImputer

imp_mean = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)

imputer = imp_mean.fit([data])

data = imputer.transform([data])

print(data)

ValueError: Input contains infinity or a value too large for dtype('float64').

We raise the error because the SimpleImputer method does not support infinite values. To solve this error, you can replace the np.inf with np.nan values as follows:

data[data==np.inf] = np.nan

imp_mean = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)

imputer = imp_mean.fit([data])

data = imputer.transform([data])

print(data)

With all infinity values replaced with NaN values, we can use the SimpleImputer to transform the data. Let’s look at the clean dataset:

[[-0.5318616   0.          0.12842066  0.          0.          0.
   1.24679674  0.09636847  0.67969774  1.2029146   0.          0.60090616
  -0.46642723  0.          1.58596659  0.47893738  1.52861316  0.
  -1.36273437  0.        ]]

Consider the case where we have multi-dimensional data with NaN and infinity values, and we want to use the SimpleImputer method. In that case, we can replace the infinite by using the Pandas replace() method as follows:

from sklearn.impute import SimpleImputer

data = [[7, 2, np.nan], 
        [4, np.nan, 6], 
        [10, 5, np.inf]]

df = pd.DataFrame(data)

df.replace([np.inf, -np.inf], np.nan, inplace=True)

data = df.to_numpy()

Then we can use the SimpleImputer to fit and transform the data. In this case, we will replace the missing values with the mean along the column where each NaN value occurs.

imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')

imp_mean.fit(data)

data = imp_mean.transform(data)

print(data)

The clean data looks like this:

[[ 7.   2.   6. ]
 [ 4.   3.5  6. ]
 [10.   5.   6. ]]

Summary

Congratulations on reading to the end of this tutorial! If you pass a NaN or an infinite value to a function, you may raise the error: ValueError: input contains nan, infinity or a value too large for dtype(‘float64’). This commonly occurs as a result of not preprocessing data before analysis. To solve this error, check your data for NaN and inf values and either remove them or replace them with real numbers.

You can only replace NaN values with the SimpleImputer method. If you try to replace infinity values with the SimpleImputer, you will raise the ValueError. Ensure that you convert all positive and negative infinity values to NaN before using the SimpleImputer.

For further reading on ValueErrors, go to the article: How to Solve Python ValueError: I/O operation on closed file.

or further reading on Scikit-learn, go to the article: How to Solve Sklearn ValueError: Unknown label type: ‘continuous’.

Go to the online courses page on Python to learn more about coding in Python for data science and machine learning.

Have fun and happy researching!

I am trying to fit my data into my model which takes numpy as input, so I feed the model with the dataframe values

stacked_averaged_models.fit(train.values, y_train1)

I am getting the following error

ValueError                                Traceback (most recent call last)
<ipython-input-145-9ba69af8df05> in <module>()
      1 X_traintrain = train.as_matrix().astype(np.float)
      2 from sklearn.metrics import r2_score
----> 3 stacked_averaged_models.fit(train.values, y_train1)
      4 stacked_train_pred = stacked_averaged_models.predict(train.values)
      5 stacked_pred = np.expm1(stacked_averaged_models.predict(test.values))

<ipython-input-140-dfca4af6e9d1> in fit(self, X, y)
     18                 instance = clone(model)
     19                 self.base_models_[i].append(instance)
---> 20                 instance.fit(X[train_index], y[train_index])
     21                 y_pred = instance.predict(X[holdout_index])
     22                 out_of_fold_predictions[holdout_index, i] = y_pred

~Anaconda3envsdeeplearninglibsite-packagessklearnpipeline.py in fit(self, X, y, **fit_params)
    248         Xt, fit_params = self._fit(X, y, **fit_params)
    249         if self._final_estimator is not None:
--> 250             self._final_estimator.fit(Xt, y, **fit_params)
    251         return self
    252 

~Anaconda3envsdeeplearninglibsite-packagessklearnlinear_modelcoordinate_descent.py in fit(self, X, y, check_input)
    705                              order='F', dtype=[np.float64, np.float32],
    706                              copy=self.copy_X and self.fit_intercept,
--> 707                              multi_output=True, y_numeric=True)
    708             y = check_array(y, order='F', copy=False, dtype=X.dtype.type,
    709                             ensure_2d=False)

~Anaconda3envsdeeplearninglibsite-packagessklearnutilsvalidation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    574     if multi_output:
    575         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,
--> 576                         dtype=None)
    577     else:
    578         y = column_or_1d(y, warn=True)

~Anaconda3envsdeeplearninglibsite-packagessklearnutilsvalidation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    451                              % (array.ndim, estimator_name))
    452         if force_all_finite:
--> 453             _assert_all_finite(array)
    454 
    455     shape_repr = _shape_repr(array.shape)

~Anaconda3envsdeeplearninglibsite-packagessklearnutilsvalidation.py in _assert_all_finite(X)
     42             and not np.isfinite(X).all()):
     43         raise ValueError("Input contains NaN, infinity"
---> 44                          " or a value too large for %r." % X.dtype)
     45 
     46 

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

I did a check on NaN and infinity, it did pass the test

X_traintrain = train.as_matrix().astype(np.float)
print(np.any(np.isnan(X_traintrain)))
print(np.all(np.isfinite(X_traintrain)))

Output:

False
True

How else can I solve, or at least, debug this?

X1      X2       X3     X4       X5    X6   X7     X8   Y1      Y2
0.64    784.00  343.00  220.50  3.50    5   0.00    0   10.56   16.67
0.62    808.50  367.50  220.50  3.50    2   0.00    0   8.60    12.07
0.62    808.50  367.50  220.50  3.50    5   0.00    0   8.50    12.04
0.98    514.50  294.00  110.25  7.00    2   0.10    1   24.58   26.47

This is few rows of my dataset

The “valueerror: input contains nan, infinity, or a value too large for dtype(‘float64’)” error message could be removed through scaling technique before modeling or getting the boolean mask back. This article describes all the causes of this error and the solutions and strategies suggested by experts.

Keep reading to learn what the answer is to fix this error.

Why Are You Getting Valueerror: Input Contains Nan, Infinity, or a Value Too Large for Dtype(‘float64’).?

You are getting this error because of using the older versions of libraries, significant errors, prediction of NaN or Inf, iterated auto_arima, affinity propagation, data mistake,

– Older Versions of Libraries

If you are working in some statistical library in Python and are getting this error, there is a high chance that you have not updated your library. Even the older versions of libraries like Pandas and Statsmodel could also cause this error.

– Large Errors

Having large errors can also be the cause of this error. If you have colossal errors, the Auto-ARIMA cannot find a reasonable solution; in that case, you might face the valueerror: input contains nan, infinity, or a value too significant for dtype(‘float64’) error.

– Prediction of Nan or Inf

Sometimes prediction of NaN and Infinite values also be the cause of that error. Especially when the prediction of NaN or Inf is in R or auto_arima.

– Iterated Auto-arima

Your auto-arima can be iterated after a group on different IDs. Let’s suppose each of your IDs has dates between 25 to 28, and the prediction is only for a single day; in that case, you can get this type of error. Your ID can face failure because of many dates, which can be non-consecutive. Additionally, that can also happen even if your Auto-arima runs on numerical arrays and have no reference to dates.

– Affinity Propagation

Sometimes this error appears while using the affinity propagation algorithm on Sklearn. This problem often occurs in the input matrix.

– Data Mistake

This error might appear because of the dataset. A mistake in data can lead you to get the error. However, the confusing thing here is that it would seem like a Sklearn, as there would be a mistake in the data.

– Having Nan and Infinity in the Input Data

One of the leading causes of this error is having the NaN and infinity in the input data. If you have Nan and inf instead of zeros and finite numbers, valueerror: input contains nan, infinity, or a value too large for dtype(‘float64’) can appear. Here you need to get a boolean mask back and tuple with i, j coordinates.

We covered several causes of toft error. Let’s see the solution to each cause.

– Older Versions of Libraries

If you are using the older versions of libraries, the simple solution to this cause would be to update all the libraries you are using.

– Large Errors

If there are significant errors in your code, you might face that error. In that case, you can try not using the Auto-Arima if the series is challenging to forecast.

– Prediction of Nan or Inf in R or Auto_arima

If you think the problem is the prediction of a NaN or Inf, you should try using a scaling technique before modeling. However, scaling should not be required because it should be tested unscaled before looking at what effects scaling provides.

If the problem is still not solved, you should try within the function, because when the modules are sequenced, there is a chance of failures during cross-validation. In addition, you can even try/accept block fails in the software-engineering pipeline sometimes.

– Iterated Auto_arima

In the case of Iterated Auto-arima, if you are facing failure because of the ID dates, make sure if any non-consecutive dates are causing the problem; if you find such a mistake, remove those non-consecutive dates, and you will no longer see that error.

– Affinity Propagation

If you are using the affinity propagation algorithm on Sklearn and having this issue, the first thing you need to do is to remove infinite values in the matrix, and for that reason, you can use the following code:

mat[np.isfinite(mat) == True]=0

This will remove the infinite values in the matrix, and you can get rid of the problem.

– Data Mistake

If you are confident that there is some mistake in the dataset, you should check your data multiple times. Make sure while conversion to float64, both of them should be finite and not NaN.

Make sure that you do not have any missing values in your dataset. To avoid this problem, you can use the imputer class as well.  You should always keep a check on the dataset. It is extremely important to process the correct data, one mistake in the dataset and you will not get the expected results.

You can clean the data set of NaA and Inf and missing cells by using the following function as well:

import pandas as pd
import numpy as np
def clean_dataset(df):
assert isinstance (df, pd.DataFrame), “df need to be a pd.DataFrame”
df.dropna(inplace=True)
indices_to _keep=~df.isin([np.nan, np.inf, -np.inf]).any(1)
return df[indices_to_keep].astype(np.float64)

– Another Way To Impute the Missing Values

You can also use the following code to get rid of infinite values. Using the following code means creating a separate function to eliminate the missing values.

df.replaces([np.inf, -np.inf], np.nan, inplace=True)

But if the problem is still not solved, then the problem might occur in the math between input and output. This math depends on some conditions. Nonetheless, there are high chances that this might be because of not fulfilling the matrix criteria. Make sure you read the documentation for the function you are using.

– Having Nan and Infinity in the Input Data

To remove the NaN and infinity in the input data, you need to get a boolean mask back with true for positions containing NaNs, and for that, you can use no.isnan(X). Note that you also need to get back a tuple with i, j coordinates of NaNs, and for that, you can use np.where(np.isnan(X)). In the final step, you need to replace NaN with zero and infinite with finite numbers, and for that, you can use the np.nan_to_num(X).

As an alternative, you can also use the sklearn.impute.SimpleImputer, that will mean or median imputations of missing values. You can also use the pandas’ pd.DataFrame(X).fillna() will allow you to fill something other than zeros.

You can use the x!=x to check the NaN. If x!=x return none, the np.isnan(X) might get failed.

Replacing the NaN values with zeros is not always the best way. If you replace the NaN values with zeros, the zeros might be introduced arbitrarily, which can lead to skewing your variables. Zero sometimes is not even an acceptable value in your variables, which means the variables will not have a true zero value.

If you want to replace the missing values with the mean values of a rolling window by using a rolling average, you can use the .rolling() for that. You can also use the module <b>missingpy</b> for more robustness.

Another solution to this error could be if you add some black rows while forming X vectors; this is how the NaNs might not get into the data frame in the first place.

FAQ Section

– What Are Nan Values?

NaN stands for not a number, and it represents the missing value in the data. Having NaN value in the data is one of the major problems. The NaN values are special floating-point values, and you can’t convert them to any other type of float. It is very necessary to remove the NaN values to get the expected results.

– How To Replace the Nan Values With Zeros in Pandas Dataframe?

You can use different functions to get rid of NaN values and replace them with zeros. You can use the fillna () function to fill Na/NaN values. The second method you can use is dataframe.replace() function used to replace the string, list, regex, dictionary, etc., in the dataframe.

Steps To Replace Nan Values:

• Use df[‘DataFrame Column’] = df[‘DataFrame Column’].fillna(0) in Pandas for one coumn.
• Then use df[‘DataFrame Column’] = df[‘DataFrame Column’].replace(np.nan,0) in numpy for one column.
• Then use df.fillna(0) in Pandas for the whole DataFrame
• Then use df.replace(np.nan, 0) in Pandas for the whole DataFrame

– What Is Dtype(‘float64’)?

The float64 is actually a Pandas type of float, and it represents numeric characters with decimals. Suppose a column contains numbers and missing values; pandas will display an error with dtype(‘float64’) if your missing value has a decimal.

– What if Your Dataset Contains No Nan or Infinite Value, and Still Throws the Dtype(‘float64’) Error?

This is the major issue where when you will check the NaN or infinite values, you would find no NaN or infinite values, but the point is, are you checking in the right way? Most beginners make this mistake and use np.all() to check if all the data are infinite, which is not the right way. You should use np.any() to help you find the NaN and infinite values; however, when you use the right method, you will definitely find the NaN and infinite values.

Conclusion

Let’s understand what we learned today:

• There could be multiple causes of that error; you should identify the cause in your case and then go for the correct solution.
• The cause of the valueerror: input contains nan, infinity, or a value too large for dtype(‘float64’) could be because of having any NaN or infinity value in the dataset, using older versions of libraries, large errors, prediction of NaN or Inf, iterated Auto_arima, affinity propagation, or any mistake in the data.
• If you have identified the cause of the error in your case, you use the solution method described above.

You have understood all the causes of this error, in addition, we have covered all the solutions for each cause. So this error should be a hurdle for you to perform with your favorite framework. Use this article as a guide and find the right solution.

При действии MinMaxScaler и при самому предсказанию с помощью рандомного леса выдаётся ошибка:

C:ProgramDataAnaconda3libsite-packagesnumpycore_methods.py:32: RuntimeWarning: invalid value encountered in reduce
return umr_sum(a, axis, dtype, out, keepdims)
ValueError Traceback (most recent call last)
in ()
2
3 scaler = MinMaxScaler()
—-> 4 x_train = scaler.fit_transform(x_train)
5 x_validation = scaler.fit_transform(x_validation)

C:ProgramDataAnaconda3libsite-packagessklearnbase.py in fit_transform(self, X, y, **fit_params)
515 if y is None:
516 # fit method of arity 1 (unsupervised transformation)
–> 517 return self.fit(X, **fit_params).transform(X)
518 else:
519 # fit method of arity 2 (supervised transformation)

C:ProgramDataAnaconda3libsite-packagessklearnpreprocessingdata.py in fit(self, X, y)
306 # Reset internal state before fitting
307 self._reset()
–> 308 return self.partial_fit(X, y)
309
310 def partial_fit(self, X, y=None):

C:ProgramDataAnaconda3libsite-packagessklearnpreprocessingdata.py in partial_fit(self, X, y)
332
333 X = check_array(X, copy=self.copy, warn_on_dtype=True,
–> 334 estimator=self, dtype=FLOAT_DTYPES)
335
336 data_min = np.min(X, axis=0)

C:ProgramDataAnaconda3libsite-packagessklearnutilsvalidation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
451 % (array.ndim, estimator_name))
452 if force_all_finite:
–> 453 _assert_all_finite(array)
454
455 shape_repr = _shape_repr(array.shape)

C:ProgramDataAnaconda3libsite-packagessklearnutilsvalidation.py in _assert_all_finite(X)
42 and not np.isfinite(X).all()):
43 raise ValueError(“Input contains NaN, infinity”
—> 44 “ or a value too large for %r.” % X.dtype)
45
46

ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’).

Не знаю, с этим делать, NaN быть не должно, так как юзал fillna(0). Чисел, которые не влазят во float64 тоже быть не должно. Понимаю, что инфы мало, но, может, кто натолкнёт на мысль, где хотя б искать проблему.