Spyder: Cannot open certain DataFrame in Variable Explorer, but can open others

[everetr]

Issue Report Checklist

• [X] Searched the issues page for similar reports
• [X] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [X] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [ ] Could not reproduce inside jupyter qtconsole (if console-related)
• [X] Tried basic troubleshooting (if a bug/error)
  
  
  
  • [X] Restarted Spyder
  
  
  • [X] Reset preferences with spyder --reset
  
  
  • [ ] Reinstalled the latest version of Anaconda
  
  
  • [X] Tried the other applicable steps from the Troubleshooting Guide
  
  
• [X] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

I create a DataFrame with the following Python3 code:

from bs4 import BeautifulSoup
import pandas as pd
from requests import get

BASE = 'https://www.njsp.org/ucr/'
URL = 'https://www.njsp.org/ucr/current-crime-data1.shtml?agree=0'

response = get(URL)
soup = BeautifulSoup(response.text, 'html.parser')

ucr_files = pd.DataFrame(
  {
  'link' : BASE + str(soup.select('.table-striped a')),
  'desc' : soup.select('.table-striped a'),
  'update' : soup.select('td:nth-child(2)')
  }
)

But I cannot open ucr_files in Variable Explorer. When I double-click it, nothing happens. I can view ucr_files in the console:

I made a small DataFrame and I can open that in variable explorer.

test1 = pd.DataFrame({
        'a': [1, 2, 3],
        'b': [4, 5, 6],
        'c': ['x', 'y', 'z']
        })

Versions

• Spyder version: 3.3.6
• Python version: 3.7.3
• Qt version: 5.9.7
• PyQt version: 5.9.2
• Operating System name/version: Ubuntu 18.04.3

Dependencies

IPython >=4.0     :  7.8.0 (OK)
cython >=0.21     :  0.29.13 (OK)
jedi >=0.9.0      :  0.15.1 (OK)
matplotlib >=2.0.0:  3.1.1 (OK)
nbconvert >=4.0   :  5.6.0 (OK)
numpy >=1.7       :  1.17.2 (OK)
pandas >=0.13.1   :  0.25.1 (OK)
psutil >=0.3      :  5.6.3 (OK)
pycodestyle >=2.3 :  2.5.0 (OK)
pyflakes >=0.6.0  :  2.1.1 (OK)
pygments >=2.0    :  2.4.2 (OK)
pylint >=0.25     :  2.3.1 (OK)
qtconsole >=4.2.0 :  4.5.5 (OK)
rope >=0.9.4      :  0.14.0 (OK)
sphinx >=0.6.6    :  2.2.0 (OK)
sympy >=0.7.3     :  1.4 (OK)

Also, if it helps, all of my packages, including Spyder and its dependencies, were installed from conda-forge.

[bcolsen]

I reproduced this in master. I do get a popup error:

Spyder was unable to retrieve the value of this variable from the console.

The error mesage was:
Could not pickle object as excessively deep recursion required.

The spelling mistake in "message" was there too.

@everetr I don't think we are going to fix this because your code is placing bs4.element.Tag objects into dataframe when I think you just want the string information.

BASE + str(soup.select('.table-striped a') This code is not going to work like you want it to. You are casting a list as a string which python probably shouldn't even let you do.

I am closing this issue, but I fixed your code for fun. Well it does what I think you wanted it to do.

from bs4 import BeautifulSoup
import pandas as pd
from requests import get

BASE = 'https://www.njsp.org/ucr/'
URL = 'https://www.njsp.org/ucr/current-crime-data1.shtml?agree=0'

response = get(URL)
soup = BeautifulSoup(response.text, 'html.parser')

ucr_files2 = pd.DataFrame(
  {
  'link' : [BASE + tag.attrs['href'] for tag in soup.select('.table-striped a')],
  'desc' : [tag.text for tag in soup.select('.table-striped a')],
  'updated' : [tag.text for tag in soup.select('td:nth-child(2)')]
  }
)

[everetr]

I appreciate the code fix.