Valores faltantes
Jeshua Romero Guadarrama- 39 Min para leer
- 01 Aug, 2021
Índice de contenido
Es común encontrar valores faltantes en los conjuntos de datos, los cuales suelen expresarse como “NaN” (“Not a Number”) o como “NA” (“Not Available”), aunque también pueden aparecer de otras maneras (“NaT” para “Not a Time”, “None”, etc.). Muchas librerías de aprendizaje automático requieren datos completos, sin valores vacíos. Por ello, es muy importante aprender a detectarlos y, sobre todo, aprender a rellenarlos o eliminarlos cuando sea necesario.
En pandas, los NaN se interpretan como valores de tipo float, por lo que una columna de enteros con al menos un NaN, se transformará en una columna de tipo float.
python
import pandas as pd
python
mydf = pd.read_csv("C:/Users/user/Desktop/dow_jones_index.data", parse_dates = True)
mydf.head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 114691279.0 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 80023895.0 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 132981863.0 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 109493077.0 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 114332562.0 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 130374108.0 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | NaN | NaN | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 45102042.0 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 25913713.0 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 38824728.0 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 51427274.0 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 39501680.0 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 43746998.0 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 28564910.0 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 |
python
mydf.dtypes
quarter int64
stock object
date object
open object
high object
low object
close object
volume int64
percent_change_price float64
percent_change_volume_over_last_wk float64
previous_weeks_volume float64
next_weeks_open object
next_weeks_close object
percent_change_next_weeks_price float64
days_to_next_dividend int64
percent_return_next_dividend float64
dtype: object
Detectar valores faltantes
¿Hay algún NaN en el DataFrame?
El siguiente código devuelve “True” si existe algún NaN en nuestro DataFrame:
python
mydf.isnull().values.any()
True
¿En qué columnas hay NaN?
Para saber en qué columnas se encuentran los NaN podemos hacer:
python
mydf.isnull().any()
quarter False
stock False
date False
open False
high False
low False
close False
volume False
percent_change_price False
percent_change_volume_over_last_wk True
previous_weeks_volume True
next_weeks_open False
next_weeks_close False
percent_change_next_weeks_price False
days_to_next_dividend False
percent_return_next_dividend False
dtype: bool
¿Cuántos NaN hay en cada columna?
Para conocer el número de NaN que hay en cada columna:
python
mydf.isnull().sum()
quarter 0
stock 0
date 0
open 0
high 0
low 0
close 0
volume 0
percent_change_price 0
percent_change_volume_over_last_wk 30
previous_weeks_volume 30
next_weeks_open 0
next_weeks_close 0
percent_change_next_weeks_price 0
days_to_next_dividend 0
percent_return_next_dividend 0
dtype: int64
¿Cuántos NaN hay en total?
python
mydf.isnull().sum().sum()
60
Extracción de filas con NaN
python
nan_rows = mydf[mydf.isnull().any(1)]
nan_rows
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | NaN | NaN | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 24 | 1 | BA | 1/7/2011 | $66.15 | $70.10 | $66.00 | $69.38 | 36258120 | 4.882840 | NaN | NaN | $69.42 | $70.07 | 0.936330 | 33 | 0.605362 |
| 36 | 1 | BAC | 1/7/2011 | $13.85 | $14.69 | $13.80 | $14.25 | 1453438639 | 2.888090 | NaN | NaN | $14.17 | $15.25 | 7.621740 | 54 | 0.070175 |
| 48 | 1 | CAT | 1/7/2011 | $94.38 | $94.81 | $92.30 | $93.73 | 24135903 | -0.688705 | NaN | NaN | $93.21 | $94.01 | 0.858277 | 11 | 0.469433 |
| 60 | 1 | CSCO | 1/7/2011 | $20.45 | $21.00 | $20.38 | $20.97 | 303545878 | 2.542790 | NaN | NaN | $20.94 | $21.21 | 1.289400 | 81 | 0.286123 |
| 72 | 1 | CVX | 1/7/2011 | $91.66 | $92.48 | $90.27 | $91.19 | 35556288 | -0.512765 | NaN | NaN | $90.95 | $92.83 | 2.067070 | 38 | 0.789560 |
| 84 | 1 | DD | 1/7/2011 | $50.05 | $50.54 | $49.28 | $49.76 | 27658789 | -0.579421 | NaN | NaN | $48.30 | $49.80 | 3.105590 | 35 | 0.823955 |
| 96 | 1 | DIS | 1/7/2011 | $37.74 | $40.00 | $37.62 | $39.45 | 72917621 | 4.531000 | NaN | NaN | $39.01 | $39.29 | 0.717765 | 336 | 1.013940 |
| 108 | 1 | GE | 1/7/2011 | $18.49 | $18.72 | $18.12 | $18.43 | 280146510 | -0.324500 | NaN | NaN | $18.61 | $18.82 | 1.128430 | 38 | 0.759631 |
| 120 | 1 | HD | 1/7/2011 | $35.20 | $35.57 | $34.18 | $34.38 | 56576860 | -2.329550 | NaN | NaN | $34.16 | $35.89 | 5.064400 | 60 | 0.727167 |
| 132 | 1 | HPQ | 1/7/2011 | $42.22 | $45.39 | $42.22 | $45.09 | 100020724 | 6.797730 | NaN | NaN | $44.86 | $46.25 | 3.098530 | 66 | 0.177423 |
| 144 | 1 | IBM | 1/7/2011 | $147.21 | $148.86 | $146.64 | $147.93 | 23492843 | 0.489097 | NaN | NaN | $147.00 | $150.00 | 2.040820 | 32 | 0.439397 |
| 156 | 1 | INTC | 1/7/2011 | $21.01 | $21.21 | $20.27 | $20.66 | 386719626 | -1.665870 | NaN | NaN | $20.71 | $21.08 | 1.786580 | 27 | 0.871249 |
| 168 | 1 | JNJ | 1/7/2011 | $62.63 | $63.54 | $62.53 | $62.60 | 57774737 | -0.047900 | NaN | NaN | $62.29 | $62.55 | 0.417402 | 49 | 0.862620 |
| 180 | 1 | JPM | 1/7/2011 | $43.00 | $44.95 | $42.64 | $43.64 | 234547885 | 1.488370 | NaN | NaN | $43.27 | $44.91 | 3.790150 | 87 | 0.572869 |
| 192 | 1 | KRFT | 1/7/2011 | $31.76 | $31.76 | $31.14 | $31.19 | 44971770 | -1.794710 | NaN | NaN | $30.91 | $31.34 | 1.391140 | 81 | 0.929785 |
| 204 | 1 | KO | 1/7/2011 | $65.88 | $65.88 | $62.56 | $62.92 | 59802189 | -4.493020 | NaN | NaN | $62.70 | $63.13 | 0.685805 | 63 | 0.746980 |
| 216 | 1 | MCD | 1/7/2011 | $77.10 | $77.59 | $73.59 | $74.37 | 85400677 | -3.540860 | NaN | NaN | $74.25 | $74.06 | -0.255892 | 49 | 0.820223 |
| 228 | 1 | MMM | 1/7/2011 | $86.75 | $87.90 | $85.63 | $86.23 | 16166921 | -0.599424 | NaN | NaN | $85.70 | $88.10 | 2.800470 | 40 | 0.637829 |
| 240 | 1 | MRK | 1/7/2011 | $36.29 | $37.35 | $35.85 | $37.35 | 72760487 | 2.920910 | NaN | NaN | $37.26 | $34.23 | -8.132040 | 63 | 1.017400 |
| 252 | 1 | MSFT | 1/7/2011 | $28.05 | $28.85 | $27.77 | $28.60 | 328646154 | 1.960780 | NaN | NaN | $28.20 | $28.30 | 0.354610 | 39 | 0.559441 |
| 264 | 1 | PFE | 1/7/2011 | $17.70 | $18.38 | $17.62 | $18.34 | 386804789 | 3.615820 | NaN | NaN | $18.22 | $18.34 | 0.658617 | 26 | 1.090510 |
| 276 | 1 | PG | 1/7/2011 | $64.39 | $65.08 | $64.00 | $64.50 | 52323352 | 0.170834 | NaN | NaN | $64.40 | $65.53 | 1.754660 | 12 | 0.744186 |
| 288 | 1 | T | 1/7/2011 | $29.68 | $30.10 | $28.66 | $28.85 | 157834347 | -2.796500 | NaN | NaN | $28.54 | $28.43 | -0.385424 | 30 | 1.490470 |
| 300 | 1 | TRV | 1/7/2011 | $56.09 | $56.34 | $52.88 | $53.33 | 20010115 | -4.920660 | NaN | NaN | $53.12 | $54.63 | 2.842620 | 60 | 0.675042 |
| 312 | 1 | UTX | 1/7/2011 | $79.19 | $79.75 | $78.40 | $79.08 | 15797841 | -0.138906 | NaN | NaN | $78.27 | $79.08 | 1.034880 | 40 | 0.543753 |
| 324 | 1 | VZ | 1/7/2011 | $36.06 | $37.70 | $35.41 | $35.93 | 141938064 | -0.360510 | NaN | NaN | $36.55 | $35.46 | -2.982220 | 89 | 1.363760 |
| 336 | 1 | WMT | 1/7/2011 | $54.23 | $55.07 | $53.76 | $54.08 | 64231179 | -0.276600 | NaN | NaN | $53.65 | $54.81 | 2.162160 | 61 | 0.684172 |
| 348 | 1 | XOM | 1/7/2011 | $73.72 | $75.90 | $73.64 | $75.59 | 101740933 | 2.536630 | NaN | NaN | $75.13 | $77.84 | 3.607080 | 52 | 0.582088 |
python
len(nan_rows)
30
Extracción de columnas con NaN
python
nan_cols = mydf.columns[mydf.isnull().any()]
nan_cols
Index(['percent_change_volume_over_last_wk', 'previous_weeks_volume'], dtype='object')
Posiciones de los NaN
python
nan_rows.index
Int64Index([ 0, 12, 24, 36, 48, 60, 72, 84, 96, 108, 120, 132, 144,
156, 168, 180, 192, 204, 216, 228, 240, 252, 264, 276, 288, 300,
312, 324, 336, 348],
dtype='int64')
Eliminar filas y/o columnas con valores faltantes
Para evitar el problema de los valores faltantes, la opción más simple consiste en eliminar las columnas que tienen valores faltantes y dejar solo aquellas que están completas. Sin embargo, este enfoque puede provocar que el modelo pierda información valiosa.
Para eliminar filas o columnas con valores faltantes, es muy útil dropna() de pandas.
Eliminar filas con, al menos, un NaN
python
mydf.dropna(how="any")
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.004240 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.521610 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.420980 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.225000 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.377620 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
720 rows × 16 columns
Eliminar filas en las que todos los valores son NaN
python
mydf.dropna(how="all")
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
Mantener filas con, al menos, un número concreto de valores no vacíos
python
len(mydf.columns)
16
python
# Mantenemos las filas con, al menos, 2 valores que no estén vacíos
mydf.dropna(thresh = 15)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.004240 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.521610 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.420980 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.225000 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.377620 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
720 rows × 16 columns
Eliminar filas que tengan NaN en ciertas columnas
python
mydf.dropna(subset = ["previous_weeks_volume"])
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.004240 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.521610 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.420980 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.225000 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.377620 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
720 rows × 16 columns
Eliminar columnas que tengan al menos un NaN
python
mydf.dropna(axis = "columns", how="any")
| quarter | stock | date | open | high | low | close | volume | percent_change_price | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 14 columns
python
mydf.dropna(axis = "columns", how="all")
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
Extracción de filas sin NaN
También puede ser interesante extraer las filas que no tienen NaN en una determinada columna, en lugar de eliminar las filas que tienen NaN.
python
mydf[mydf["percent_change_volume_over_last_wk"].notna()]
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.004240 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.521610 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.420980 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.225000 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.377620 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
720 rows × 16 columns
Relleno de valores faltantes
El método de imputación rellena los valores faltantes con algún valor, como por ejemplo la media artmética de los valores de la columna. Aunque no es perfecto, este método es generalmente más efectivo que eliminar columnas completas.
Para rellenar valores faltantes, es muy útil fillna() de pandas.
Relleno con un valor constante
python
# Relleno de todos los valores faltantes del df con texto
mydf.fillna(value = "No sabe / no contesta", inplace = False)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | No sabe / no contesta | No sabe / no contesta | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.38022 | 2.39656e+08 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.025 | 2.42963e+08 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.3555 | 1.38428e+08 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.98745 | 1.51379e+08 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.3557 | 8.67588e+07 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221 | 6.82309e+07 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.5085 | 7.86163e+07 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.81195 | 9.23808e+07 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.0642 | 1.00521e+08 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
python
mydf
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
python
# Relleno de todos los valores faltantes del df con un escalar
mydf.fillna(value = 20.)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | 20.000000 | 20.0 | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
python
# Relleno de los valores faltantes de una columna concreta
mydf["percent_change_volume_over_last_wk"].fillna(value = "Valor faltante")
0 Valor faltante
1 1.38022
2 -43.025
3 9.3555
4 1.98745
...
745 -21.3557
746 15.221
747 17.5085
748 8.81195
749 18.0642
Name: percent_change_volume_over_last_wk, Length: 750, dtype: object
python
mydf.dtypes
quarter int64
stock object
date object
open object
high object
low object
close object
volume int64
percent_change_price float64
percent_change_volume_over_last_wk float64
previous_weeks_volume float64
next_weeks_open object
next_weeks_close object
percent_change_next_weeks_price float64
days_to_next_dividend int64
percent_return_next_dividend float64
dtype: object
python
mydf.describe()
| quarter | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|
| count | 750.000000 | 7.500000e+02 | 750.000000 | 720.000000 | 7.200000e+02 | 750.000000 | 750.000000 | 750.000000 |
| mean | 1.520000 | 1.175478e+08 | 0.050262 | 5.593627 | 1.173876e+08 | 0.238468 | 52.525333 | 0.691826 |
| std | 0.499933 | 1.584381e+08 | 2.517809 | 40.543478 | 1.592322e+08 | 2.679538 | 46.335098 | 0.305482 |
| min | 1.000000 | 9.718851e+06 | -15.422900 | -61.433175 | 9.718851e+06 | -15.422900 | 0.000000 | 0.065574 |
| 25% | 1.000000 | 3.086624e+07 | -1.288053 | -19.804284 | 3.067832e+07 | -1.222067 | 24.000000 | 0.534549 |
| 50% | 2.000000 | 5.306088e+07 | 0.000000 | 0.512586 | 5.294556e+07 | 0.101193 | 47.000000 | 0.681067 |
| 75% | 2.000000 | 1.327218e+08 | 1.650888 | 21.800622 | 1.333230e+08 | 1.845562 | 69.000000 | 0.854291 |
| max | 2.000000 | 1.453439e+09 | 9.882230 | 327.408924 | 1.453439e+09 | 9.882230 | 336.000000 | 1.564210 |
python
mydf["previous_weeks_volume"].mean()
117387644.83472222
python
# Relleno de los valores faltantes de cada columna con su media respectiva
mydf.apply(lambda x: x.fillna(x.mean()))
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in _ensure_numeric(x)
1426 try:
-> 1427 x = float(x)
1428 except ValueError:
ValueError: could not convert string to float: 'AAAAAAAAAAAAAAAAAAAAAAAAAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPBABABABABABABABABABABABABACBACBACBACBACBACBACBACBACBACBACBACCATCATCATCATCATCATCATCATCATCATCATCATCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXDDDDDDDDDDDDDDDDDDDDDDDDDISDISDISDISDISDISDISDISDISDISDISDISGEGEGEGEGEGEGEGEGEGEGEGEHDHDHDHDHDHDHDHDHDHDHDHDHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKOKOKOKOKOKOKOKOKOKOKOKOMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPGPGPGPGPGPGPGPGPGPGPGPGTTTTTTTTTTTTTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXVZVZVZVZVZVZVZVZVZVZVZVZWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMAAAAAAAAAAAAAAAAAAAAAAAAAAAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPBABABABABABABABABABABABABABACBACBACBACBACBACBACBACBACBACBACBACBACCATCATCATCATCATCATCATCATCATCATCATCATCATCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXDDDDDDDDDDDDDDDDDDDDDDDDDDDISDISDISDISDISDISDISDISDISDISDISDISDISGEGEGEGEGEGEGEGEGEGEGEGEGEHDHDHDHDHDHDHDHDHDHDHDHDHDHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKOKOKOKOKOKOKOKOKOKOKOKOKOMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPGPGPGPGPGPGPGPGPGPGPGPGPGTTTTTTTTTTTTTTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXVZVZVZVZVZVZVZVZVZVZVZVZVZWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOM'
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in _ensure_numeric(x)
1430 try:
-> 1431 x = complex(x)
1432 except ValueError as err:
ValueError: complex() arg is a malformed string
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
<ipython-input-90-b49ed0a9ea6a> in <module>
1 # Relleno de los valores faltantes de cada columna con su media respectiva
----> 2 mydf.apply(lambda x: x.fillna(x.mean()))
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\frame.py in apply(self, func, axis, raw, result_type, args, **kwds)
7539 kwds=kwds,
7540 )
-> 7541 return op.get_result()
7542
7543 def applymap(self, func) -> "DataFrame":
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\apply.py in get_result(self)
178 return self.apply_raw()
179
--> 180 return self.apply_standard()
181
182 def apply_empty_result(self):
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\apply.py in apply_standard(self)
253
254 def apply_standard(self):
--> 255 results, res_index = self.apply_series_generator()
256
257 # wrap results
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\apply.py in apply_series_generator(self)
282 for i, v in enumerate(series_gen):
283 # ignore SettingWithCopy here in case the user mutates
--> 284 results[i] = self.f(v)
285 if isinstance(results[i], ABCSeries):
286 # If we have a view on v, we need to make a copy because
<ipython-input-90-b49ed0a9ea6a> in <lambda>(x)
1 # Relleno de los valores faltantes de cada columna con su media respectiva
----> 2 mydf.apply(lambda x: x.fillna(x.mean()))
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\generic.py in stat_func(self, axis, skipna, level, numeric_only, **kwargs)
11454 if level is not None:
11455 return self._agg_by_level(name, axis=axis, level=level, skipna=skipna)
> 11456 return self._reduce(
11457 func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
11458 )
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\series.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
4234 )
4235 with np.errstate(all="ignore"):
-> 4236 return op(delegate, skipna=skipna, **kwds)
4237
4238 def _reindex_indexer(self, new_index, indexer, copy):
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in _f(*args, **kwargs)
69 try:
70 with np.errstate(invalid="ignore"):
---> 71 return f(*args, **kwargs)
72 except ValueError as e:
73 # we want to transform an object array
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in f(values, axis, skipna, **kwds)
127 result = alt(values, axis=axis, skipna=skipna, **kwds)
128 else:
--> 129 result = alt(values, axis=axis, skipna=skipna, **kwds)
130
131 return result
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in nanmean(values, axis, skipna, mask)
561 dtype_count = dtype
562 count = _get_counts(values.shape, mask, axis, dtype=dtype_count)
--> 563 the_sum = _ensure_numeric(values.sum(axis, dtype=dtype_sum))
564
565 if axis is not None and getattr(the_sum, "ndim", False):
~\anaconda3\envs\python-385\lib\site-packages\pandas\core\nanops.py in _ensure_numeric(x)
1432 except ValueError as err:
1433 # e.g. "foo"
-> 1434 raise TypeError(f"Could not convert {x} to numeric") from err
1435 return x
1436
TypeError: Could not convert AAAAAAAAAAAAAAAAAAAAAAAAAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPBABABABABABABABABABABABABACBACBACBACBACBACBACBACBACBACBACBACCATCATCATCATCATCATCATCATCATCATCATCATCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXDDDDDDDDDDDDDDDDDDDDDDDDDISDISDISDISDISDISDISDISDISDISDISDISGEGEGEGEGEGEGEGEGEGEGEGEHDHDHDHDHDHDHDHDHDHDHDHDHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKOKOKOKOKOKOKOKOKOKOKOKOMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPGPGPGPGPGPGPGPGPGPGPGPGTTTTTTTTTTTTTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXVZVZVZVZVZVZVZVZVZVZVZVZWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMAAAAAAAAAAAAAAAAAAAAAAAAAAAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPAXPBABABABABABABABABABABABABABACBACBACBACBACBACBACBACBACBACBACBACBACCATCATCATCATCATCATCATCATCATCATCATCATCATCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCSCOCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXCVXDDDDDDDDDDDDDDDDDDDDDDDDDDDISDISDISDISDISDISDISDISDISDISDISDISDISGEGEGEGEGEGEGEGEGEGEGEGEGEHDHDHDHDHDHDHDHDHDHDHDHDHDHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQHPQIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMIBMINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCINTCJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJNJJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMJPMKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKRFTKOKOKOKOKOKOKOKOKOKOKOKOKOMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMCDMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMRKMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTMSFTPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPFEPGPGPGPGPGPGPGPGPGPGPGPGPGTTTTTTTTTTTTTTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVTRVUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXUTXVZVZVZVZVZVZVZVZVZVZVZVZVZWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTWMTXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOMXOM to numeric
python
mydf
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
python
mydf.mean()
quarter 1.520000e+00
volume 1.175478e+08
percent_change_price 5.026241e-02
percent_change_volume_over_last_wk 5.593627e+00
previous_weeks_volume 1.173876e+08
percent_change_next_weeks_price 2.384681e-01
days_to_next_dividend 5.252533e+01
percent_return_next_dividend 6.918256e-01
dtype: float64
python
mydf.fillna(mydf.mean())
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | 5.593627 | 1.173876e+08 | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 2.396556e+08 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 2.429634e+08 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 1.384285e+08 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 1.513792e+08 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 8.675882e+07 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 6.823086e+07 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 7.861630e+07 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 9.238084e+07 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 1.005214e+08 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
Relleno con el valor anterior
Con datos de series de tiempo, el uso de “pad” / “ffill” es extremadamente común para que el “último valor conocido” esté disponible en cada punto de tiempo.
python
mydf.fillna(method = "pad").head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 114691279.0 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 80023895.0 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 132981863.0 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 109493077.0 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 114332562.0 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 130374108.0 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | -26.710607 | 130374108.0 | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 45102042.0 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 25913713.0 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 38824728.0 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 51427274.0 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 39501680.0 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 43746998.0 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 28564910.0 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 |
Relleno con el valor posterior
Esta opción se puede indicar tanto con “bfill” como con “backfill”:
python
mydf.fillna(method = "bfill")
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | 1.380223 | 239655616.0 | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 86758820.0 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 68230855.0 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 78616295.0 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 92380844.0 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 100521400.0 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
Relleno mediante un diccionario de valores
python
mydf.head(2)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.42849 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.47066 | 19 | 0.187852 |
python
mydf.fillna(value = {"percent_change_volume_over_last_wk": 0.,
"previous_weeks_volume": "None"})
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | 0.000000 | None | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223 | 2.39656e+08 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.024959 | 2.42963e+08 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500 | 1.38428e+08 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987452 | 1.51379e+08 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.355713 | 8.67588e+07 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.221032 | 6.82309e+07 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.508519 | 7.86163e+07 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.811952 | 9.23808e+07 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.064204 | 1.00521e+08 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
Relleno estableciendo un límite
python
mydf.head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 114691279.0 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 80023895.0 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 132981863.0 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 109493077.0 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 114332562.0 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 130374108.0 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | NaN | NaN | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 45102042.0 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 25913713.0 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 38824728.0 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 51427274.0 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 39501680.0 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 43746998.0 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 28564910.0 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 |
python
mydf.fillna(value = {"percent_change_volume_over_last_wk": 0.,
"previous_weeks_volume": "None"},
limit = 1).head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | 0.000000 | None | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 2.39656e+08 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 2.42963e+08 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 1.38428e+08 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 1.51379e+08 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 1.54388e+08 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 1.14691e+08 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 8.00239e+07 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 1.32982e+08 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 1.09493e+08 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 1.14333e+08 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 1.30374e+08 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | NaN | NaN | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 4.5102e+07 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 2.59137e+07 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 3.88247e+07 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 5.14273e+07 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 3.95017e+07 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 4.3747e+07 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 2.85649e+07 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 |
Relleno con un valor interpolado
Los objetos Series y DataFrame tienen interpolate() que, por defecto, realiza una interpolación lineal en los puntos de datos faltantes.
Si se trata de una serie temporal que está creciendo a un ritmo creciente, puede ser más apropiado method=‘quadratic’.
Si hay valores aproximados a una función de distribución acumulativa, entonces method=‘pchip’debería funcionar bien.
Para completar los valores faltantes con un trazado suave, considere method=‘akima’.
Al interpolar a través de una aproximación polinómica o spline, también se debe especificar el grado u orden de la aproximación:
python
mydf.interpolate(method= "linear").head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 114691279.0 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 80023895.0 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 132981863.0 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 109493077.0 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 114332562.0 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 130374108.0 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | -34.627433 | 87738075.0 | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 45102042.0 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 25913713.0 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 38824728.0 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 51427274.0 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 39501680.0 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 43746998.0 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 28564910.0 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 |
Imputación con información adicional
La imputación de valores faltantes suele funcionar bastante bien, pero los valores introducidos no dejan de ser valores aproximados a los que podrían ser en realidad. Si, de alguna manera, le indicamos al modelo qué valores son originales y cuáles han sido imputados, le estaremos proporcionando un extra de información que en algunos casos podrá aprovechar para hacer predicciones más precisas. Una forma habitual de aportar este extra de información consiste en crear una nueva columna que indique qué valores son “TRUE” (originales) y cuáles son “FALSE” (imputados).
python
mydf.head(5)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NAN | NAN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223028 | 239655616 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.02495926 | 242963398 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500109 | 138428495 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987451735 | 151379173 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
python
mydf.dtypes
quarter int64
stock object
date object
open object
high object
low object
close object
volume int64
percent_change_price float64
percent_change_volume_over_last_wk object
previous_weeks_volume object
next_weeks_open object
next_weeks_close object
percent_change_next_weeks_price float64
days_to_next_dividend int64
percent_return_next_dividend float64
dtype: object
python
mydf.isnull().sum()
quarter 0
stock 0
date 0
open 0
high 0
low 0
close 0
volume 0
percent_change_price 0
percent_change_volume_over_last_wk 30
previous_weeks_volume 30
next_weeks_open 0
next_weeks_close 0
percent_change_next_weeks_price 0
days_to_next_dividend 0
percent_return_next_dividend 0
dtype: int64
python
mydf.replace("NAN", np.nan, inplace = True)
python
mydf
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.79267 | NaN | NaN | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.42849 | 1.380223028 | 239655616 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.47066 | -43.02495926 | 242963398 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.63831 | 9.355500109 | 138428495 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.93325 | 1.987451735 | 151379173 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 745 | 2 | XOM | 5/27/2011 | $80.22 | $82.63 | $80.07 | $82.63 | 68230855 | 3.00424 | -21.35571346 | 86758820 | $83.28 | $81.18 | -2.521610 | 75 | 0.568801 |
| 746 | 2 | XOM | 6/3/2011 | $83.28 | $83.75 | $80.18 | $81.18 | 78616295 | -2.52161 | 15.2210316 | 68230855 | $80.93 | $79.78 | -1.420980 | 68 | 0.578960 |
| 747 | 2 | XOM | 6/10/2011 | $80.93 | $81.87 | $79.72 | $79.78 | 92380844 | -1.42098 | 17.50851907 | 78616295 | $80.00 | $79.02 | -1.225000 | 61 | 0.589120 |
| 748 | 2 | XOM | 6/17/2011 | $80.00 | $80.82 | $78.33 | $79.02 | 100521400 | -1.22500 | 8.8119524 | 92380844 | $78.65 | $76.78 | -2.377620 | 54 | 0.594786 |
| 749 | 2 | XOM | 6/24/2011 | $78.65 | $81.12 | $76.78 | $76.78 | 118679791 | -2.37762 | 18.06420424 | 100521400 | $76.88 | $82.01 | 6.672740 | 47 | 0.612139 |
750 rows × 16 columns
python
mydf.fillna("VALOR_FALTANTE", inplace = True)
python
mydf.head(20)
| quarter | stock | date | open | high | low | close | volume | percent_change_price | percent_change_volume_over_last_wk | previous_weeks_volume | next_weeks_open | next_weeks_close | percent_change_next_weeks_price | days_to_next_dividend | percent_return_next_dividend | Pista_modelo | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | AA | 1/7/2011 | $15.82 | $16.72 | $15.78 | $16.42 | 239655616 | 3.792670 | 0.000000 | 0.0 | $16.71 | $15.97 | -4.428490 | 26 | 0.182704 | FALSO |
| 1 | 1 | AA | 1/14/2011 | $16.71 | $16.71 | $15.64 | $15.97 | 242963398 | -4.428490 | 1.380223 | 239655616.0 | $16.19 | $15.79 | -2.470660 | 19 | 0.187852 | VERDADERO |
| 2 | 1 | AA | 1/21/2011 | $16.19 | $16.38 | $15.60 | $15.79 | 138428495 | -2.470660 | -43.024959 | 242963398.0 | $15.87 | $16.13 | 1.638310 | 12 | 0.189994 | VERDADERO |
| 3 | 1 | AA | 1/28/2011 | $15.87 | $16.63 | $15.82 | $16.13 | 151379173 | 1.638310 | 9.355500 | 138428495.0 | $16.18 | $17.14 | 5.933250 | 5 | 0.185989 | VERDADERO |
| 4 | 1 | AA | 2/4/2011 | $16.18 | $17.39 | $16.18 | $17.14 | 154387761 | 5.933250 | 1.987452 | 151379173.0 | $17.33 | $17.37 | 0.230814 | 97 | 0.175029 | VERDADERO |
| 5 | 1 | AA | 2/11/2011 | $17.33 | $17.48 | $16.97 | $17.37 | 114691279 | 0.230814 | -25.712195 | 154387761.0 | $17.39 | $17.28 | -0.632547 | 90 | 0.172712 | VERDADERO |
| 6 | 1 | AA | 2/18/2011 | $17.39 | $17.68 | $17.28 | $17.28 | 80023895 | -0.632547 | -30.226696 | 114691279.0 | $16.98 | $16.68 | -1.766780 | 83 | 0.173611 | VERDADERO |
| 7 | 1 | AA | 2/25/2011 | $16.98 | $17.15 | $15.96 | $16.68 | 132981863 | -1.766780 | 66.177694 | 80023895.0 | $16.81 | $16.58 | -1.368230 | 76 | 0.179856 | VERDADERO |
| 8 | 1 | AA | 3/4/2011 | $16.81 | $16.94 | $16.13 | $16.58 | 109493077 | -1.368230 | -17.663150 | 132981863.0 | $16.58 | $16.03 | -3.317250 | 69 | 0.180941 | VERDADERO |
| 9 | 1 | AA | 3/11/2011 | $16.58 | $16.75 | $15.42 | $16.03 | 114332562 | -3.317250 | 4.419900 | 109493077.0 | $15.95 | $16.11 | 1.003130 | 62 | 0.187149 | VERDADERO |
| 10 | 1 | AA | 3/18/2011 | $15.95 | $16.33 | $15.43 | $16.11 | 130374108 | 1.003130 | 14.030601 | 114332562.0 | $16.38 | $17.09 | 4.334550 | 55 | 0.186220 | VERDADERO |
| 11 | 1 | AA | 3/25/2011 | $16.38 | $17.24 | $16.26 | $17.09 | 95550392 | 4.334550 | -26.710607 | 130374108.0 | $17.13 | $17.47 | 1.984820 | 48 | 0.175541 | VERDADERO |
| 12 | 1 | AXP | 1/7/2011 | $43.30 | $45.60 | $43.11 | $44.36 | 45102042 | 2.448040 | 0.000000 | 0.0 | $44.20 | $46.25 | 4.638010 | 89 | 0.405771 | FALSO |
| 13 | 1 | AXP | 1/14/2011 | $44.20 | $46.25 | $44.01 | $46.25 | 25913713 | 4.638010 | -42.544258 | 45102042.0 | $46.03 | $46.00 | -0.065175 | 82 | 0.389189 | VERDADERO |
| 14 | 1 | AXP | 1/21/2011 | $46.03 | $46.71 | $44.71 | $46.00 | 38824728 | -0.065175 | 49.823099 | 25913713.0 | $46.05 | $43.86 | -4.755700 | 75 | 0.391304 | VERDADERO |
| 15 | 1 | AXP | 1/28/2011 | $46.05 | $46.27 | $43.42 | $43.86 | 51427274 | -4.755700 | 32.460101 | 38824728.0 | $44.13 | $43.82 | -0.702470 | 68 | 0.410397 | VERDADERO |
| 16 | 1 | AXP | 2/4/2011 | $44.13 | $44.23 | $43.15 | $43.82 | 39501680 | -0.702470 | -23.189240 | 51427274.0 | $43.96 | $46.75 | 6.346680 | 61 | 0.410771 | VERDADERO |
| 17 | 1 | AXP | 2/11/2011 | $43.96 | $46.79 | $43.88 | $46.75 | 43746998 | 6.346680 | 10.747183 | 39501680.0 | $46.42 | $45.53 | -1.917280 | 54 | 0.385027 | VERDADERO |
| 18 | 1 | AXP | 2/18/2011 | $46.42 | $46.93 | $45.53 | $45.53 | 28564910 | -1.917280 | -34.704297 | 43746998.0 | $44.94 | $43.53 | -3.137520 | 47 | 0.395344 | VERDADERO |
| 19 | 1 | AXP | 2/25/2011 | $44.94 | $45.12 | $43.01 | $43.53 | 39654146 | -3.137520 | 38.821183 | 28564910.0 | $43.73 | $43.72 | -0.022868 | 40 | 0.413508 | VERDADERO |
python
mydf["Pista_modelo"] = "VERDADERO"
python
mydf.loc[mydf["percent_change_volume_over_last_wk"] == "VALOR_FALTANTE", "Pista_modelo"] = "FALSO"
python
mydf.replace("VALOR_FALTANTE", np.nan, inplace = True)
python
mydf.fillna(0., inplace = True)
Aunque este es un buen enfoque para tratar los valores faltantes, la columna de información adicional no siempre consigue mejorar los resultados del modelo de forma significativa.
Puede ocurrir que aparentemente tengamos valores faltantes, pero que en realidad no lo sean. Puede que se trate simplemente de strings “NaN”. En este caso, necesitamos reemplazarlos por verdaderos valores faltantes.