假设我正在Jupyter/Ipython笔记本中进行大量耗时计算的数据分析。然后，由于某种原因，我必须关闭Jupyter本地服务器，但是我希望稍后返回继续进行分析，而不必再次进行所有耗时的计算。
%storemagic，应该是“轻量级pickle”。然而，您必须手动存储变量，如下所示：#在ipython/nb会话中foo = "一个虚拟字符串"%store foo%store -r foo # r用于刷新print(foo) # "一个虚拟字符串"pip install dill）很好地回答了你的问题。dill.dump_session来保存一个Notebook会话：import dill
dill.dump_session('notebook_env.db')
dill.load_session来恢复一个Notebook会话：import dill
dill.load_session('notebook_env.db')
我更愿意评论而不是将此作为真正的答案提供，但我需要更多的声望才能进行评论。
您可以以系统化的方式存储大多数类似数据的变量。 我通常会将所有数据框，数组等存储在pandas.HDFStore中。 在笔记本的开头，声明：
backup = pd.HDFStore('backup.h5')
然后将你生成的任何新变量存储起来
backup['var1'] = var1
最后，可能做一件好事
backup.close()
在关闭服务器之前。下次您想继续使用笔记本电脑时:
backup = pd.HDFStore('backup.h5')
var1 = backup['var1']
说实话，我也更喜欢IPython笔记本中的内置功能。但是你无法通过这种方式保存所有内容（例如对象、连接），并且在有太多样板代码时难以保持笔记本的组织性。
%%cache longcalc.pkl var1 var2 var3
var1 = longcalculation()
....
自从首次发布这个答案后，我制作了一个pypi包https://pypi.org/project/jupyter-save-load-vars。可以通过以下方式进行安装
pip install jupyter-save-load-vars
由于dill.dump_session在无法pickle化的对象上失败，并且显然无法配置为简单地忽略这些对象，因此我编写了一对静态函数savevars(file)和loadvars()，它们位于this ne1 library中，用于我们的神经形态工程jupyter笔记本类芯片练习。
这种方法基于有用的帖子Get locals from calling namespace in Python
from ne1 import savevars,loadvars
a=1
b=[2,3]
c='string'
o=(i for i in []) # make generator that cannot be pickled
savevars('testvars')
del a,b,c
loadvars('testvars')
print(a)
print(b)
print(c)
输出：
[INFO]: 2023-10-02 08:28:32,878 - NE1 - saved to testvars.dill variables [ a b c ] (File "/home/ne1/CoACH-labs/ne1.py", line 132, in savevars)
[WARNING]: 2023-10-02 08:28:32,879 - NE1 - could not pickle: ['o'] (File "/home/ne1/CoACH-labs/ne1.py", line 134, in savevars)
[INFO]: 2023-10-02 08:28:32,881 - NE1 - from testvars.dill loaded variables ['a', 'b', 'c'] (File "/home/ne1/CoACH-labs/ne1.py", line 158, in loadvars)
1
[2, 3]
string
def savevars(filename):
"""
saves all local variables to a file with dill
:param filename: the name of the file. The suffix .dill is added if there is not a suffix already.
"""
if filename is None:
log.error('you must supply a filename')
return
from pathlib import Path
p=Path(filename)
if p.suffix=='': # if suffix is missing add .dill
p = p.parent / (p.name + _DILL)
import inspect,dill
locals=None
frame = inspect.currentframe().f_back
try:
locals = frame.f_locals
finally:
del frame
if locals is None: return
data={}
could_not_pickle=[]
from types import ModuleType
s=f'saved to {p} variables [ '
for k,v in locals.items():
# don't try to pickle any pyplane objects
if k.startswith('_') or k=='tmp' or k=='In' or k=='Out' or hasattr(v, '__call__') \
or isinstance(v,ModuleType) or isinstance(v,logging.Logger):
continue
try:
if not dill.pickles(v):
could_not_pickle.append(k)
continue
except:
could_not_pickle.append(k)
continue
s=s+k+' '
data[k]=v
s=s+']'
try:
with open(p,'wb') as f:
try:
dill.dump(data,f)
log.info(f'{s}')
if len(could_not_pickle)>0:
log.warning(f'could not pickle: {could_not_pickle}')
except TypeError as e:
log.error(f'\n Error: {e}')
except Exception as e:
log.error(f'could not save data to {p}')
def loadvars(filename):
""" Loads variables from file into the current workspace
:param filename: the dill file to load from, e.g. lab1. The suffix .dill is added automatically unless there is already a suffix.
This function loads the variables found in filename into the parent workspace.
"""
import dill
from pathlib import Path
p=Path(filename)
if p.suffix=='': # if suffix is missing add .dill
p = p.parent / (p.name + _DILL)
if not p.exists:
log.error(f'{p} does not exist')
return
try:
with open(p,'rb') as f:
data=dill.load(f)
log.info(f'from {p} loaded variables {list(data.keys())}')
import inspect
try:
frame = inspect.currentframe().f_back # get the workspace frame (jupyter workspace frame)
locals = frame.f_locals # get its local variable dict
for k in data:
try:
locals[k] = data[k] # set a value in it
except Exception as e:
log.error(f'could not set variable {k}')
finally:
del frame
except Exception as e:
log.error(f'could not load; got {e}')
import logging
# general logger. Produces nice output format with live hyperlinks for pycharm users
# to use it, just call log=get_logger() at the top of your python file
# all these loggers share the same logger name 'Control_Toolkit'
_LOGGING_LEVEL = logging.DEBUG # usually INFO is good
class CustomFormatter(logging.Formatter):
"""Logging Formatter to add colors and count warning / errors"""
# see https://dev59.com/wnRC5IYBdhLWcg3wMd5S#7995762
# \x1b[ (ESC[) is the CSI introductory sequence for ANSI https://en.wikipedia.org/wiki/ANSI_escape_code
# The control sequence CSI n m, named Select Graphic Rendition (SGR), sets display attributes.
grey = "\x1b[2;37m" # 2 faint, 37 gray
yellow = "\x1b[33;21m"
cyan = "\x1b[0;36m" # 0 normal 36 cyan
green = "\x1b[31;21m" # dark green
red = "\x1b[31;21m" # bold red
bold_red = "\x1b[31;1m"
light_blue = "\x1b[1;36m"
blue = "\x1b[1;34m"
reset = "\x1b[0m"
# File "{file}", line {max(line, 1)}'.replace("\\", "/")
format = '[%(levelname)s]: %(asctime)s - %(name)s - %(message)s (File "%(pathname)s", line %(lineno)d, in %(funcName)s)'
FORMATS = {
logging.DEBUG: grey + format + reset,
logging.INFO: cyan + format + reset,
logging.WARNING: red + format + reset,
logging.ERROR: bold_red + format + reset,
logging.CRITICAL: bold_red + format + reset
}
def format(self, record):
log_fmt = self.FORMATS.get(record.levelno)
formatter = logging.Formatter(log_fmt)
return formatter.format(record).replace("\\", "/") #replace \ with / for pycharm links
def get_logger():
""" Use get_logger to define a logger with useful color output and info and warning turned on according to the global LOGGING_LEVEL.
:returns: the logger.
"""
# logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logger = logging.getLogger('NE1') # tobi changed so all have same name so we can uniformly affect all of them
logger.setLevel(_LOGGING_LEVEL)
# create console handler if this logger does not have handler yet
if len(logger.handlers)==0:
ch = logging.StreamHandler()
ch.setFormatter(CustomFormatter())
logger.addHandler(ch)
return logger
log=get_logger()