1 # -*- coding: utf-8 -*- 2 """ 3 Sphinx directive to support embedded IPython code. 4 5 IPython provides an extension for `Sphinx <http://www.sphinx-doc.org/>`_ to 6 highlight and run code. 7 8 This directive allows pasting of entire interactive IPython sessions, prompts 9 and all, and their code will actually get re-executed at doc build time, with 10 all prompts renumbered sequentially. It also allows you to input code as a pure 11 python input by giving the argument python to the directive. The output looks 12 like an interactive ipython section. 13 14 Here is an example of how the IPython directive can 15 **run** python code, at build time. 16 17 .. ipython:: 18 19 In [1]: 1+1 20 21 In [1]: import datetime 22 ...: datetime.date.fromisoformat('2022-02-22') 23 24 It supports IPython construct that plain 25 Python does not understand (like magics): 26 27 .. ipython:: 28 29 In [0]: import time 30 31 In [0]: %pdoc time.sleep 32 33 This will also support top-level async when using IPython 7.0+ 34 35 .. ipython:: 36 37 In [2]: import asyncio 38 ...: print('before') 39 ...: await asyncio.sleep(1) 40 ...: print('after') 41 42 43 The namespace will persist across multiple code chucks, Let's define a variable: 44 45 .. ipython:: 46 47 In [0]: who = "World" 48 49 And now say hello: 50 51 .. ipython:: 52 53 In [0]: print('Hello,', who) 54 55 If the current section raises an exception, you can add the ``:okexcept:`` flag 56 to the current block, otherwise the build will fail. 57 58 .. ipython:: 59 :okexcept: 60 61 In [1]: 1/0 62 63 IPython Sphinx directive module 64 =============================== 65 66 To enable this directive, simply list it in your Sphinx ``conf.py`` file 67 (making sure the directory where you placed it is visible to sphinx, as is 68 needed for all Sphinx directives). For example, to enable syntax highlighting 69 and the IPython directive:: 70 71 extensions = ['IPython.sphinxext.ipython_console_highlighting', 72 'IPython.sphinxext.ipython_directive'] 73 74 The IPython directive outputs code-blocks with the language 'ipython'. So 75 if you do not have the syntax highlighting extension enabled as well, then 76 all rendered code-blocks will be uncolored. By default this directive assumes 77 that your prompts are unchanged IPython ones, but this can be customized. 78 The configurable options that can be placed in conf.py are: 79 80 ipython_savefig_dir: 81 The directory in which to save the figures. This is relative to the 82 Sphinx source directory. The default is `html_static_path`. 83 ipython_rgxin: 84 The compiled regular expression to denote the start of IPython input 85 lines. The default is ``re.compile('In \\[(\\d+)\\]:\\s?(.*)\\s*')``. You 86 shouldn't need to change this. 87 ipython_warning_is_error: [default to True] 88 Fail the build if something unexpected happen, for example if a block raise 89 an exception but does not have the `:okexcept:` flag. The exact behavior of 90 what is considered strict, may change between the sphinx directive version. 91 ipython_rgxout: 92 The compiled regular expression to denote the start of IPython output 93 lines. The default is ``re.compile('Out\\[(\\d+)\\]:\\s?(.*)\\s*')``. You 94 shouldn't need to change this. 95 ipython_promptin: 96 The string to represent the IPython input prompt in the generated ReST. 97 The default is ``'In [%d]:'``. This expects that the line numbers are used 98 in the prompt. 99 ipython_promptout: 100 The string to represent the IPython prompt in the generated ReST. The 101 default is ``'Out [%d]:'``. This expects that the line numbers are used 102 in the prompt. 103 ipython_mplbackend: 104 The string which specifies if the embedded Sphinx shell should import 105 Matplotlib and set the backend. The value specifies a backend that is 106 passed to `matplotlib.use()` before any lines in `ipython_execlines` are 107 executed. If not specified in conf.py, then the default value of 'agg' is 108 used. To use the IPython directive without matplotlib as a dependency, set 109 the value to `None`. It may end up that matplotlib is still imported 110 if the user specifies so in `ipython_execlines` or makes use of the 111 @savefig pseudo decorator. 112 ipython_execlines: 113 A list of strings to be exec'd in the embedded Sphinx shell. Typical 114 usage is to make certain packages always available. Set this to an empty 115 list if you wish to have no imports always available. If specified in 116 ``conf.py`` as `None`, then it has the effect of making no imports available. 117 If omitted from conf.py altogether, then the default value of 118 ['import numpy as np', 'import matplotlib.pyplot as plt'] is used. 119 ipython_holdcount 120 When the @suppress pseudo-decorator is used, the execution count can be 121 incremented or not. The default behavior is to hold the execution count, 122 corresponding to a value of `True`. Set this to `False` to increment 123 the execution count after each suppressed command. 124 125 As an example, to use the IPython directive when `matplotlib` is not available, 126 one sets the backend to `None`:: 127 128 ipython_mplbackend = None 129 130 An example usage of the directive is: 131 132 .. code-block:: rst 133 134 .. ipython:: 135 136 In [1]: x = 1 137 138 In [2]: y = x**2 139 140 In [3]: print(y) 141 142 See http://matplotlib.org/sampledoc/ipython_directive.html for additional 143 documentation. 144 145 Pseudo-Decorators 146 ================= 147 148 Note: Only one decorator is supported per input. If more than one decorator 149 is specified, then only the last one is used. 150 151 In addition to the Pseudo-Decorators/options described at the above link, 152 several enhancements have been made. The directive will emit a message to the 153 console at build-time if code-execution resulted in an exception or warning. 154 You can suppress these on a per-block basis by specifying the :okexcept: 155 or :okwarning: options: 156 157 .. code-block:: rst 158 159 .. ipython:: 160 :okexcept: 161 :okwarning: 162 163 In [1]: 1/0 164 In [2]: # raise warning. 165 166 To Do 167 ===== 168 169 - Turn the ad-hoc test() function into a real test suite. 170 - Break up ipython-specific functionality from matplotlib stuff into better 171 separated code. 172 173 """ 174 175 # Authors 176 # ======= 177 # 178 # - John D Hunter: original author. 179 # - Fernando Perez: refactoring, documentation, cleanups, port to 0.11. 180 # - VáclavŠmilauer <eudoxos-AT-arcig.cz>: Prompt generalizations. 181 # - Skipper Seabold, refactoring, cleanups, pure python addition 182 183 #----------------------------------------------------------------------------- 184 # Imports 185 #----------------------------------------------------------------------------- 186 187 # Stdlib 188 import atexit 189 import errno 190 import os 191 import pathlib 192 import re 193 import sys 194 import tempfile 195 import ast 196 import warnings 197 import shutil 198 from io import StringIO 199 from typing import Any, Dict, Set 200 201 # Third-party 202 from docutils.parsers.rst import directives 203 from docutils.parsers.rst import Directive 204 from sphinx.util import logging 205 206 # Our own 207 from traitlets.config import Config 208 from IPython import InteractiveShell 209 from IPython.core.profiledir import ProfileDir 210 211 use_matplotlib = False 212 try: 213 import matplotlib 214 use_matplotlib = True 215 except Exception: 216 pass 217 218 #----------------------------------------------------------------------------- 219 # Globals 220 #----------------------------------------------------------------------------- 221 # for tokenizing blocks 222 COMMENT, INPUT, OUTPUT = range(3) 223 224 PSEUDO_DECORATORS = ["suppress", "verbatim", "savefig", "doctest"] 225 226 #----------------------------------------------------------------------------- 227 # Functions and class declarations 228 #----------------------------------------------------------------------------- 229 230 def block_parser(part, rgxin, rgxout, fmtin, fmtout): 231 """ 232 part is a string of ipython text, comprised of at most one 233 input, one output, comments, and blank lines. The block parser 234 parses the text into a list of:: 235 236 blocks = [ (TOKEN0, data0), (TOKEN1, data1), ...] 237 238 where TOKEN is one of [COMMENT | INPUT | OUTPUT ] and 239 data is, depending on the type of token:: 240 241 COMMENT : the comment string 242 243 INPUT: the (DECORATOR, INPUT_LINE, REST) where 244 DECORATOR: the input decorator (or None) 245 INPUT_LINE: the input as string (possibly multi-line) 246 REST : any stdout generated by the input line (not OUTPUT) 247 248 OUTPUT: the output string, possibly multi-line 249 250 """ 251 block = [] 252 lines = part.split('\n') 253 N = len(lines) 254 i = 0 255 decorator = None 256 while 1: 257 258 if i==N: 259 # nothing left to parse -- the last line 260 break 261 262 line = lines[i] 263 i += 1 264 line_stripped = line.strip() 265 if line_stripped.startswith('#'): 266 block.append((COMMENT, line)) 267 continue 268 269 if any( 270 line_stripped.startswith("@" + pseudo_decorator) 271 for pseudo_decorator in PSEUDO_DECORATORS 272 ): 273 if decorator: 274 raise RuntimeError( 275 "Applying multiple pseudo-decorators on one line is not supported" 276 ) 277 else: 278 decorator = line_stripped 279 continue 280 281 # does this look like an input line? 282 matchin = rgxin.match(line) 283 if matchin: 284 lineno, inputline = int(matchin.group(1)), matchin.group(2) 285 286 # the ....: continuation string 287 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2)) 288 Nc = len(continuation) 289 # input lines can continue on for more than one line, if 290 # we have a '\' line continuation char or a function call 291 # echo line 'print'. The input line can only be 292 # terminated by the end of the block or an output line, so 293 # we parse out the rest of the input line if it is 294 # multiline as well as any echo text 295 296 rest = [] 297 while i<N: 298 299 # look ahead; if the next line is blank, or a comment, or 300 # an output line, we're done 301 302 nextline = lines[i] 303 matchout = rgxout.match(nextline) 304 # print("nextline=%s, continuation=%s, starts=%s"%(nextline, continuation, nextline.startswith(continuation))) 305 if matchout or nextline.startswith('#'): 306 break 307 elif nextline.startswith(continuation): 308 # The default ipython_rgx* treat the space following the colon as optional. 309 # However, If the space is there we must consume it or code 310 # employing the cython_magic extension will fail to execute. 311 # 312 # This works with the default ipython_rgx* patterns, 313 # If you modify them, YMMV. 314 nextline = nextline[Nc:] 315 if nextline and nextline[0] == ' ': 316 nextline = nextline[1:] 317 318 inputline += '\n' + nextline 319 else: 320 rest.append(nextline) 321 i+= 1 322 323 block.append((INPUT, (decorator, inputline, '\n'.join(rest)))) 324 continue 325 326 # if it looks like an output line grab all the text to the end 327 # of the block 328 matchout = rgxout.match(line) 329 if matchout: 330 lineno, output = int(matchout.group(1)), matchout.group(2) 331 if i<N-1: 332 output = '\n'.join([output] + lines[i:]) 333 334 block.append((OUTPUT, output)) 335 break 336 337 return block 338 339 340 class EmbeddedSphinxShell(object): 341 """An embedded IPython instance to run inside Sphinx""" 342 343 def __init__(self, exec_lines=None): 344 345 self.cout = StringIO() 346 347 if exec_lines is None: 348 exec_lines = [] 349 350 # Create config object for IPython 351 config = Config() 352 config.HistoryManager.hist_file = ':memory:' 353 config.InteractiveShell.autocall = False 354 config.InteractiveShell.autoindent = False 355 config.InteractiveShell.colors = 'NoColor' 356 357 # create a profile so instance history isn't saved 358 tmp_profile_dir = tempfile.mkdtemp(prefix='profile_') 359 profname = 'auto_profile_sphinx_build' 360 pdir = os.path.join(tmp_profile_dir,profname) 361 profile = ProfileDir.create_profile_dir(pdir) 362 363 # Create and initialize global ipython, but don't start its mainloop. 364 # This will persist across different EmbeddedSphinxShell instances. 365 IP = InteractiveShell.instance(config=config, profile_dir=profile) 366 atexit.register(self.cleanup) 367 368 # Store a few parts of IPython we'll need. 369 self.IP = IP 370 self.user_ns = self.IP.user_ns 371 self.user_global_ns = self.IP.user_global_ns 372 373 self.input = '' 374 self.output = '' 375 self.tmp_profile_dir = tmp_profile_dir 376 377 self.is_verbatim = False 378 self.is_doctest = False 379 self.is_suppress = False 380 381 # Optionally, provide more detailed information to shell. 382 # this is assigned by the SetUp method of IPythonDirective 383 # to point at itself. 384 # 385 # So, you can access handy things at self.directive.state 386 self.directive = None 387 388 # on the first call to the savefig decorator, we'll import 389 # pyplot as plt so we can make a call to the plt.gcf().savefig 390 self._pyplot_imported = False 391 392 # Prepopulate the namespace. 393 for line in exec_lines: 394 self.process_input_line(line, store_history=False) 395 396 def cleanup(self): 397 shutil.rmtree(self.tmp_profile_dir, ignore_errors=True) 398 399 def clear_cout(self): 400 self.cout.seek(0) 401 self.cout.truncate(0) 402 403 def process_input_line(self, line, store_history): 404 return self.process_input_lines([line], store_history=store_history) 405 406 def process_input_lines(self, lines, store_history=True): 407 """process the input, capturing stdout""" 408 stdout = sys.stdout 409 source_raw = '\n'.join(lines) 410 try: 411 sys.stdout = self.cout 412 self.IP.run_cell(source_raw, store_history=store_history) 413 finally: 414 sys.stdout = stdout 415 416 def process_image(self, decorator): 417 """ 418 # build out an image directive like 419 # .. image:: somefile.png 420 # :width 4in 421 # 422 # from an input like 423 # savefig somefile.png width=4in 424 """ 425 savefig_dir = self.savefig_dir 426 source_dir = self.source_dir 427 saveargs = decorator.split(' ') 428 filename = saveargs[1] 429 # insert relative path to image file in source 430 # as absolute path for Sphinx 431 # sphinx expects a posix path, even on Windows 432 path = pathlib.Path(savefig_dir, filename) 433 outfile = '/' + path.relative_to(source_dir).as_posix() 434 435 imagerows = ['.. image:: %s' % outfile] 436 437 for kwarg in saveargs[2:]: 438 arg, val = kwarg.split('=') 439 arg = arg.strip() 440 val = val.strip() 441 imagerows.append(' :%s: %s'%(arg, val)) 442 443 image_file = os.path.basename(outfile) # only return file name 444 image_directive = '\n'.join(imagerows) 445 return image_file, image_directive 446 447 # Callbacks for each type of token 448 def process_input(self, data, input_prompt, lineno): 449 """ 450 Process data block for INPUT token. 451 452 """ 453 decorator, input, rest = data 454 image_file = None 455 image_directive = None 456 457 is_verbatim = decorator=='@verbatim' or self.is_verbatim 458 is_doctest = (decorator is not None and \ 459 decorator.startswith('@doctest')) or self.is_doctest 460 is_suppress = decorator=='@suppress' or self.is_suppress 461 is_okexcept = decorator=='@okexcept' or self.is_okexcept 462 is_okwarning = decorator=='@okwarning' or self.is_okwarning 463 is_savefig = decorator is not None and \ 464 decorator.startswith('@savefig') 465 466 input_lines = input.split('\n') 467 if len(input_lines) > 1: 468 if input_lines[-1] != "": 469 input_lines.append('') # make sure there's a blank line 470 # so splitter buffer gets reset 471 472 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2)) 473 474 if is_savefig: 475 image_file, image_directive = self.process_image(decorator) 476 477 ret = [] 478 is_semicolon = False 479 480 # Hold the execution count, if requested to do so. 481 if is_suppress and self.hold_count: 482 store_history = False 483 else: 484 store_history = True 485 486 # Note: catch_warnings is not thread safe 487 with warnings.catch_warnings(record=True) as ws: 488 if input_lines[0].endswith(';'): 489 is_semicolon = True 490 #for i, line in enumerate(input_lines): 491 492 # process the first input line 493 if is_verbatim: 494 self.process_input_lines(['']) 495 self.IP.execution_count += 1 # increment it anyway 496 else: 497 # only submit the line in non-verbatim mode 498 self.process_input_lines(input_lines, store_history=store_history) 499 500 if not is_suppress: 501 for i, line in enumerate(input_lines): 502 if i == 0: 503 formatted_line = '%s %s'%(input_prompt, line) 504 else: 505 formatted_line = '%s %s'%(continuation, line) 506 ret.append(formatted_line) 507 508 if not is_suppress and len(rest.strip()) and is_verbatim: 509 # The "rest" is the standard output of the input. This needs to be 510 # added when in verbatim mode. If there is no "rest", then we don't 511 # add it, as the new line will be added by the processed output. 512 ret.append(rest) 513 514 # Fetch the processed output. (This is not the submitted output.) 515 self.cout.seek(0) 516 processed_output = self.cout.read() 517 if not is_suppress and not is_semicolon: 518 # 519 # In IPythonDirective.run, the elements of `ret` are eventually 520 # combined such that '' entries correspond to newlines. So if 521 # `processed_output` is equal to '', then the adding it to `ret` 522 # ensures that there is a blank line between consecutive inputs 523 # that have no outputs, as in: 524 # 525 # In [1]: x = 4 526 # 527 # In [2]: x = 5 528 # 529 # When there is processed output, it has a '\n' at the tail end. So 530 # adding the output to `ret` will provide the necessary spacing 531 # between consecutive input/output blocks, as in: 532 # 533 # In [1]: x 534 # Out[1]: 5 535 # 536 # In [2]: x 537 # Out[2]: 5 538 # 539 # When there is stdout from the input, it also has a '\n' at the 540 # tail end, and so this ensures proper spacing as well. E.g.: 541 # 542 # In [1]: print(x) 543 # 5 544 # 545 # In [2]: x = 5 546 # 547 # When in verbatim mode, `processed_output` is empty (because 548 # nothing was passed to IP. Sometimes the submitted code block has 549 # an Out[] portion and sometimes it does not. When it does not, we 550 # need to ensure proper spacing, so we have to add '' to `ret`. 551 # However, if there is an Out[] in the submitted code, then we do 552 # not want to add a newline as `process_output` has stuff to add. 553 # The difficulty is that `process_input` doesn't know if 554 # `process_output` will be called---so it doesn't know if there is 555 # Out[] in the code block. The requires that we include a hack in 556 # `process_block`. See the comments there. 557 # 558 ret.append(processed_output) 559 elif is_semicolon: 560 # Make sure there is a newline after the semicolon. 561 ret.append('') 562 563 # context information 564 filename = "Unknown" 565 lineno = 0 566 if self.directive.state: 567 filename = self.directive.state.document.current_source 568 lineno = self.directive.state.document.current_line 569 570 # Use sphinx logger for warnings 571 logger = logging.getLogger(__name__) 572 573 # output any exceptions raised during execution to stdout 574 # unless :okexcept: has been specified. 575 if not is_okexcept and ( 576 ("Traceback" in processed_output) or ("SyntaxError" in processed_output) 577 ): 578 s = "\n>>>" + ("-" * 73) + "\n" 579 s += "Exception in %s at block ending on line %s\n" % (filename, lineno) 580 s += "Specify :okexcept: as an option in the ipython:: block to suppress this message\n" 581 s += processed_output + "\n" 582 s += "<<<" + ("-" * 73) 583 logger.warning(s) 584 if self.warning_is_error: 585 raise RuntimeError( 586 "Unexpected exception in `{}` line {}".format(filename, lineno) 587 ) 588 589 # output any warning raised during execution to stdout 590 # unless :okwarning: has been specified. 591 if not is_okwarning: 592 for w in ws: 593 s = "\n>>>" + ("-" * 73) + "\n" 594 s += "Warning in %s at block ending on line %s\n" % (filename, lineno) 595 s += "Specify :okwarning: as an option in the ipython:: block to suppress this message\n" 596 s += ("-" * 76) + "\n" 597 s += warnings.formatwarning( 598 w.message, w.category, w.filename, w.lineno, w.line 599 ) 600 s += "<<<" + ("-" * 73) 601 logger.warning(s) 602 if self.warning_is_error: 603 raise RuntimeError( 604 "Unexpected warning in `{}` line {}".format(filename, lineno) 605 ) 606 607 self.clear_cout() 608 return (ret, input_lines, processed_output, 609 is_doctest, decorator, image_file, image_directive) 610 611 612 def process_output(self, data, output_prompt, input_lines, output, 613 is_doctest, decorator, image_file): 614 """ 615 Process data block for OUTPUT token. 616 617 """ 618 # Recall: `data` is the submitted output, and `output` is the processed 619 # output from `input_lines`. 620 621 TAB = ' ' * 4 622 623 if is_doctest and output is not None: 624 625 found = output # This is the processed output 626 found = found.strip() 627 submitted = data.strip() 628 629 if self.directive is None: 630 source = 'Unavailable' 631 content = 'Unavailable' 632 else: 633 source = self.directive.state.document.current_source 634 content = self.directive.content 635 # Add tabs and join into a single string. 636 content = '\n'.join([TAB + line for line in content]) 637 638 # Make sure the output contains the output prompt. 639 ind = found.find(output_prompt) 640 if ind < 0: 641 e = ('output does not contain output prompt\n\n' 642 'Document source: {0}\n\n' 643 'Raw content: \n{1}\n\n' 644 'Input line(s):\n{TAB}{2}\n\n' 645 'Output line(s):\n{TAB}{3}\n\n') 646 e = e.format(source, content, '\n'.join(input_lines), 647 repr(found), TAB=TAB) 648 raise RuntimeError(e) 649 found = found[len(output_prompt):].strip() 650 651 # Handle the actual doctest comparison. 652 if decorator.strip() == '@doctest': 653 # Standard doctest 654 if found != submitted: 655 e = ('doctest failure\n\n' 656 'Document source: {0}\n\n' 657 'Raw content: \n{1}\n\n' 658 'On input line(s):\n{TAB}{2}\n\n' 659 'we found output:\n{TAB}{3}\n\n' 660 'instead of the expected:\n{TAB}{4}\n\n') 661 e = e.format(source, content, '\n'.join(input_lines), 662 repr(found), repr(submitted), TAB=TAB) 663 raise RuntimeError(e) 664 else: 665 self.custom_doctest(decorator, input_lines, found, submitted) 666 667 # When in verbatim mode, this holds additional submitted output 668 # to be written in the final Sphinx output. 669 # https://github.com/ipython/ipython/issues/5776 670 out_data = [] 671 672 is_verbatim = decorator=='@verbatim' or self.is_verbatim 673 if is_verbatim and data.strip(): 674 # Note that `ret` in `process_block` has '' as its last element if 675 # the code block was in verbatim mode. So if there is no submitted 676 # output, then we will have proper spacing only if we do not add 677 # an additional '' to `out_data`. This is why we condition on 678 # `and data.strip()`. 679 680 # The submitted output has no output prompt. If we want the 681 # prompt and the code to appear, we need to join them now 682 # instead of adding them separately---as this would create an 683 # undesired newline. How we do this ultimately depends on the 684 # format of the output regex. I'll do what works for the default 685 # prompt for now, and we might have to adjust if it doesn't work 686 # in other cases. Finally, the submitted output does not have 687 # a trailing newline, so we must add it manually. 688 out_data.append("{0} {1}\n".format(output_prompt, data)) 689 690 return out_data 691 692 def process_comment(self, data): 693 """Process data fPblock for COMMENT token.""" 694 if not self.is_suppress: 695 return [data] 696 697 def save_image(self, image_file): 698 """ 699 Saves the image file to disk. 700 """ 701 self.ensure_pyplot() 702 command = 'plt.gcf().savefig("%s")'%image_file 703 # print('SAVEFIG', command) # dbg 704 self.process_input_line('bookmark ipy_thisdir', store_history=False) 705 self.process_input_line('cd -b ipy_savedir', store_history=False) 706 self.process_input_line(command, store_history=False) 707 self.process_input_line('cd -b ipy_thisdir', store_history=False) 708 self.process_input_line('bookmark -d ipy_thisdir', store_history=False) 709 self.clear_cout() 710 711 def process_block(self, block): 712 """ 713 process block from the block_parser and return a list of processed lines 714 """ 715 ret = [] 716 output = None 717 input_lines = None 718 lineno = self.IP.execution_count 719 720 input_prompt = self.promptin % lineno 721 output_prompt = self.promptout % lineno 722 image_file = None 723 image_directive = None 724 725 found_input = False 726 for token, data in block: 727 if token == COMMENT: 728 out_data = self.process_comment(data) 729 elif token == INPUT: 730 found_input = True 731 (out_data, input_lines, output, is_doctest, 732 decorator, image_file, image_directive) = \ 733 self.process_input(data, input_prompt, lineno) 734 elif token == OUTPUT: 735 if not found_input: 736 737 TAB = ' ' * 4 738 linenumber = 0 739 source = 'Unavailable' 740 content = 'Unavailable' 741 if self.directive: 742 linenumber = self.directive.state.document.current_line 743 source = self.directive.state.document.current_source 744 content = self.directive.content 745 # Add tabs and join into a single string. 746 content = '\n'.join([TAB + line for line in content]) 747 748 e = ('\n\nInvalid block: Block contains an output prompt ' 749 'without an input prompt.\n\n' 750 'Document source: {0}\n\n' 751 'Content begins at line {1}: \n\n{2}\n\n' 752 'Problematic block within content: \n\n{TAB}{3}\n\n') 753 e = e.format(source, linenumber, content, block, TAB=TAB) 754 755 # Write, rather than include in exception, since Sphinx 756 # will truncate tracebacks. 757 sys.stdout.write(e) 758 raise RuntimeError('An invalid block was detected.') 759 out_data = \ 760 self.process_output(data, output_prompt, input_lines, 761 output, is_doctest, decorator, 762 image_file) 763 if out_data: 764 # Then there was user submitted output in verbatim mode. 765 # We need to remove the last element of `ret` that was 766 # added in `process_input`, as it is '' and would introduce 767 # an undesirable newline. 768 assert(ret[-1] == '') 769 del ret[-1] 770 771 if out_data: 772 ret.extend(out_data) 773 774 # save the image files 775 if image_file is not None: 776 self.save_image(image_file) 777 778 return ret, image_directive 779 780 def ensure_pyplot(self): 781 """ 782 Ensures that pyplot has been imported into the embedded IPython shell. 783 784 Also, makes sure to set the backend appropriately if not set already. 785 786 """ 787 # We are here if the @figure pseudo decorator was used. Thus, it's 788 # possible that we could be here even if python_mplbackend were set to 789 # `None`. That's also strange and perhaps worthy of raising an 790 # exception, but for now, we just set the backend to 'agg'. 791 792 if not self._pyplot_imported: 793 if 'matplotlib.backends' not in sys.modules: 794 # Then ipython_matplotlib was set to None but there was a 795 # call to the @figure decorator (and ipython_execlines did 796 # not set a backend). 797 #raise Exception("No backend was set, but @figure was used!") 798 import matplotlib 799 matplotlib.use('agg') 800 801 # Always import pyplot into embedded shell. 802 self.process_input_line('import matplotlib.pyplot as plt', 803 store_history=False) 804 self._pyplot_imported = True 805 806 def process_pure_python(self, content): 807 """ 808 content is a list of strings. it is unedited directive content 809 810 This runs it line by line in the InteractiveShell, prepends 811 prompts as needed capturing stderr and stdout, then returns 812 the content as a list as if it were ipython code 813 """ 814 output = [] 815 savefig = False # keep up with this to clear figure 816 multiline = False # to handle line continuation 817 multiline_start = None 818 fmtin = self.promptin 819 820 ct = 0 821 822 for lineno, line in enumerate(content): 823 824 line_stripped = line.strip() 825 if not len(line): 826 output.append(line) 827 continue 828 829 # handle pseudo-decorators, whilst ensuring real python decorators are treated as input 830 if any( 831 line_stripped.startswith("@" + pseudo_decorator) 832 for pseudo_decorator in PSEUDO_DECORATORS 833 ): 834 output.extend([line]) 835 if 'savefig' in line: 836 savefig = True # and need to clear figure 837 continue 838 839 # handle comments 840 if line_stripped.startswith('#'): 841 output.extend([line]) 842 continue 843 844 # deal with lines checking for multiline 845 continuation = u' %s:'% ''.join(['.']*(len(str(ct))+2)) 846 if not multiline: 847 modified = u"%s %s" % (fmtin % ct, line_stripped) 848 output.append(modified) 849 ct += 1 850 try: 851 ast.parse(line_stripped) 852 output.append(u'') 853 except Exception: # on a multiline 854 multiline = True 855 multiline_start = lineno 856 else: # still on a multiline 857 modified = u'%s %s' % (continuation, line) 858 output.append(modified) 859 860 # if the next line is indented, it should be part of multiline 861 if len(content) > lineno + 1: 862 nextline = content[lineno + 1] 863 if len(nextline) - len(nextline.lstrip()) > 3: 864 continue 865 try: 866 mod = ast.parse( 867 '\n'.join(content[multiline_start:lineno+1])) 868 if isinstance(mod.body[0], ast.FunctionDef): 869 # check to see if we have the whole function 870 for element in mod.body[0].body: 871 if isinstance(element, ast.Return): 872 multiline = False 873 else: 874 output.append(u'') 875 multiline = False 876 except Exception: 877 pass 878 879 if savefig: # clear figure if plotted 880 self.ensure_pyplot() 881 self.process_input_line('plt.clf()', store_history=False) 882 self.clear_cout() 883 savefig = False 884 885 return output 886 887 def custom_doctest(self, decorator, input_lines, found, submitted): 888 """ 889 Perform a specialized doctest. 890 891 """ 892 from .custom_doctests import doctests 893 894 args = decorator.split() 895 doctest_type = args[1] 896 if doctest_type in doctests: 897 doctests[doctest_type](self, args, input_lines, found, submitted) 898 else: 899 e = "Invalid option to @doctest: {0}".format(doctest_type) 900 raise Exception(e) 901 902 903 class IPythonDirective(Directive): 904 905 has_content: bool = True 906 required_arguments: int = 0 907 optional_arguments: int = 4 # python, suppress, verbatim, doctest 908 final_argumuent_whitespace: bool = True 909 option_spec: Dict[str, Any] = { 910 "python": directives.unchanged, 911 "suppress": directives.flag, 912 "verbatim": directives.flag, 913 "doctest": directives.flag, 914 "okexcept": directives.flag, 915 "okwarning": directives.flag, 916 } 917 918 shell = None 919 920 seen_docs: Set = set() 921 922 def get_config_options(self): 923 # contains sphinx configuration variables 924 config = self.state.document.settings.env.config 925 926 # get config variables to set figure output directory 927 savefig_dir = config.ipython_savefig_dir 928 source_dir = self.state.document.settings.env.srcdir 929 savefig_dir = os.path.join(source_dir, savefig_dir) 930 931 # get regex and prompt stuff 932 rgxin = config.ipython_rgxin 933 rgxout = config.ipython_rgxout 934 warning_is_error= config.ipython_warning_is_error 935 promptin = config.ipython_promptin 936 promptout = config.ipython_promptout 937 mplbackend = config.ipython_mplbackend 938 exec_lines = config.ipython_execlines 939 hold_count = config.ipython_holdcount 940 941 return (savefig_dir, source_dir, rgxin, rgxout, 942 promptin, promptout, mplbackend, exec_lines, hold_count, warning_is_error) 943 944 def setup(self): 945 # Get configuration values. 946 (savefig_dir, source_dir, rgxin, rgxout, promptin, promptout, 947 mplbackend, exec_lines, hold_count, warning_is_error) = self.get_config_options() 948 949 try: 950 os.makedirs(savefig_dir) 951 except OSError as e: 952 if e.errno != errno.EEXIST: 953 raise 954 955 if self.shell is None: 956 # We will be here many times. However, when the 957 # EmbeddedSphinxShell is created, its interactive shell member 958 # is the same for each instance. 959 960 if mplbackend and 'matplotlib.backends' not in sys.modules and use_matplotlib: 961 import matplotlib 962 matplotlib.use(mplbackend) 963 964 # Must be called after (potentially) importing matplotlib and 965 # setting its backend since exec_lines might import pylab. 966 self.shell = EmbeddedSphinxShell(exec_lines) 967 968 # Store IPython directive to enable better error messages 969 self.shell.directive = self 970 971 # reset the execution count if we haven't processed this doc 972 #NOTE: this may be borked if there are multiple seen_doc tmp files 973 #check time stamp? 974 if not self.state.document.current_source in self.seen_docs: 975 self.shell.IP.history_manager.reset() 976 self.shell.IP.execution_count = 1 977 self.seen_docs.add(self.state.document.current_source) 978 979 # and attach to shell so we don't have to pass them around 980 self.shell.rgxin = rgxin 981 self.shell.rgxout = rgxout 982 self.shell.promptin = promptin 983 self.shell.promptout = promptout 984 self.shell.savefig_dir = savefig_dir 985 self.shell.source_dir = source_dir 986 self.shell.hold_count = hold_count 987 self.shell.warning_is_error = warning_is_error 988 989 # setup bookmark for saving figures directory 990 self.shell.process_input_line( 991 'bookmark ipy_savedir "%s"' % savefig_dir, store_history=False 992 ) 993 self.shell.clear_cout() 994 995 return rgxin, rgxout, promptin, promptout 996 997 def teardown(self): 998 # delete last bookmark 999 self.shell.process_input_line('bookmark -d ipy_savedir', 1000 store_history=False) 1001 self.shell.clear_cout() 1002 1003 def run(self): 1004 debug = False 1005 1006 #TODO, any reason block_parser can't be a method of embeddable shell 1007 # then we wouldn't have to carry these around 1008 rgxin, rgxout, promptin, promptout = self.setup() 1009 1010 options = self.options 1011 self.shell.is_suppress = 'suppress' in options 1012 self.shell.is_doctest = 'doctest' in options 1013 self.shell.is_verbatim = 'verbatim' in options 1014 self.shell.is_okexcept = 'okexcept' in options 1015 self.shell.is_okwarning = 'okwarning' in options 1016 1017 # handle pure python code 1018 if 'python' in self.arguments: 1019 content = self.content 1020 self.content = self.shell.process_pure_python(content) 1021 1022 # parts consists of all text within the ipython-block. 1023 # Each part is an input/output block. 1024 parts = '\n'.join(self.content).split('\n\n') 1025 1026 lines = ['.. code-block:: ipython', ''] 1027 figures = [] 1028 1029 # Use sphinx logger for warnings 1030 logger = logging.getLogger(__name__) 1031 1032 for part in parts: 1033 block = block_parser(part, rgxin, rgxout, promptin, promptout) 1034 if len(block): 1035 rows, figure = self.shell.process_block(block) 1036 for row in rows: 1037 lines.extend([' {0}'.format(line) 1038 for line in row.split('\n')]) 1039 1040 if figure is not None: 1041 figures.append(figure) 1042 else: 1043 message = 'Code input with no code at {}, line {}'\ 1044 .format( 1045 self.state.document.current_source, 1046 self.state.document.current_line) 1047 if self.shell.warning_is_error: 1048 raise RuntimeError(message) 1049 else: 1050 logger.warning(message) 1051 1052 for figure in figures: 1053 lines.append('') 1054 lines.extend(figure.split('\n')) 1055 lines.append('') 1056 1057 if len(lines) > 2: 1058 if debug: 1059 print('\n'.join(lines)) 1060 else: 1061 # This has to do with input, not output. But if we comment 1062 # these lines out, then no IPython code will appear in the 1063 # final output. 1064 self.state_machine.insert_input( 1065 lines, self.state_machine.input_lines.source(0)) 1066 1067 # cleanup 1068 self.teardown() 1069 1070 return [] 1071 1072 # Enable as a proper Sphinx directive 1073 def setup(app): 1074 setup.app = app 1075 1076 app.add_directive('ipython', IPythonDirective) 1077 app.add_config_value('ipython_savefig_dir', 'savefig', 'env') 1078 app.add_config_value('ipython_warning_is_error', True, 'env') 1079 app.add_config_value('ipython_rgxin', 1080 re.compile(r'In \[(\d+)\]:\s?(.*)\s*'), 'env') 1081 app.add_config_value('ipython_rgxout', 1082 re.compile(r'Out\[(\d+)\]:\s?(.*)\s*'), 'env') 1083 app.add_config_value('ipython_promptin', 'In [%d]:', 'env') 1084 app.add_config_value('ipython_promptout', 'Out[%d]:', 'env') 1085 1086 # We could just let matplotlib pick whatever is specified as the default 1087 # backend in the matplotlibrc file, but this would cause issues if the 1088 # backend didn't work in headless environments. For this reason, 'agg' 1089 # is a good default backend choice. 1090 app.add_config_value('ipython_mplbackend', 'agg', 'env') 1091 1092 # If the user sets this config value to `None`, then EmbeddedSphinxShell's 1093 # __init__ method will treat it as []. 1094 execlines = ['import numpy as np'] 1095 if use_matplotlib: 1096 execlines.append('import matplotlib.pyplot as plt') 1097 app.add_config_value('ipython_execlines', execlines, 'env') 1098 1099 app.add_config_value('ipython_holdcount', True, 'env') 1100 1101 metadata = {'parallel_read_safe': True, 'parallel_write_safe': True} 1102 return metadata 1103 1104 # Simple smoke test, needs to be converted to a proper automatic test. 1105 def test(): 1106 1107 examples = [ 1108 r""" 1109 In [9]: pwd 1110 Out[9]: '/home/jdhunter/py4science/book' 1111 1112 In [10]: cd bookdata/ 1113 /home/jdhunter/py4science/book/bookdata 1114 1115 In [2]: from pylab import * 1116 1117 In [2]: ion() 1118 1119 In [3]: im = imread('stinkbug.png') 1120 1121 @savefig mystinkbug.png width=4in 1122 In [4]: imshow(im) 1123 Out[4]: <matplotlib.image.AxesImage object at 0x39ea850> 1124 1125 """, 1126 r""" 1127 1128 In [1]: x = 'hello world' 1129 1130 # string methods can be 1131 # used to alter the string 1132 @doctest 1133 In [2]: x.upper() 1134 Out[2]: 'HELLO WORLD' 1135 1136 @verbatim 1137 In [3]: x.st<TAB> 1138 x.startswith x.strip 1139 """, 1140 r""" 1141 1142 In [130]: url = 'http://ichart.finance.yahoo.com/table.csv?s=CROX\ 1143 .....: &d=9&e=22&f=2009&g=d&a=1&br=8&c=2006&ignore=.csv' 1144 1145 In [131]: print url.split('&') 1146 ['http://ichart.finance.yahoo.com/table.csv?s=CROX', 'd=9', 'e=22', 'f=2009', 'g=d', 'a=1', 'b=8', 'c=2006', 'ignore=.csv'] 1147 1148 In [60]: import urllib 1149 1150 """, 1151 r"""\ 1152 1153 In [133]: import numpy.random 1154 1155 @suppress 1156 In [134]: numpy.random.seed(2358) 1157 1158 @doctest 1159 In [135]: numpy.random.rand(10,2) 1160 Out[135]: 1161 array([[ 0.64524308, 0.59943846], 1162 [ 0.47102322, 0.8715456 ], 1163 [ 0.29370834, 0.74776844], 1164 [ 0.99539577, 0.1313423 ], 1165 [ 0.16250302, 0.21103583], 1166 [ 0.81626524, 0.1312433 ], 1167 [ 0.67338089, 0.72302393], 1168 [ 0.7566368 , 0.07033696], 1169 [ 0.22591016, 0.77731835], 1170 [ 0.0072729 , 0.34273127]]) 1171 1172 """, 1173 1174 r""" 1175 In [106]: print x 1176 jdh 1177 1178 In [109]: for i in range(10): 1179 .....: print i 1180 .....: 1181 .....: 1182 0 1183 1 1184 2 1185 3 1186 4 1187 5 1188 6 1189 7 1190 8 1191 9 1192 """, 1193 1194 r""" 1195 1196 In [144]: from pylab import * 1197 1198 In [145]: ion() 1199 1200 # use a semicolon to suppress the output 1201 @savefig test_hist.png width=4in 1202 In [151]: hist(np.random.randn(10000), 100); 1203 1204 1205 @savefig test_plot.png width=4in 1206 In [151]: plot(np.random.randn(10000), 'o'); 1207 """, 1208 1209 r""" 1210 # use a semicolon to suppress the output 1211 In [151]: plt.clf() 1212 1213 @savefig plot_simple.png width=4in 1214 In [151]: plot([1,2,3]) 1215 1216 @savefig hist_simple.png width=4in 1217 In [151]: hist(np.random.randn(10000), 100); 1218 1219 """, 1220 r""" 1221 # update the current fig 1222 In [151]: ylabel('number') 1223 1224 In [152]: title('normal distribution') 1225 1226 1227 @savefig hist_with_text.png 1228 In [153]: grid(True) 1229 1230 @doctest float 1231 In [154]: 0.1 + 0.2 1232 Out[154]: 0.3 1233 1234 @doctest float 1235 In [155]: np.arange(16).reshape(4,4) 1236 Out[155]: 1237 array([[ 0, 1, 2, 3], 1238 [ 4, 5, 6, 7], 1239 [ 8, 9, 10, 11], 1240 [12, 13, 14, 15]]) 1241 1242 In [1]: x = np.arange(16, dtype=float).reshape(4,4) 1243 1244 In [2]: x[0,0] = np.inf 1245 1246 In [3]: x[0,1] = np.nan 1247 1248 @doctest float 1249 In [4]: x 1250 Out[4]: 1251 array([[ inf, nan, 2., 3.], 1252 [ 4., 5., 6., 7.], 1253 [ 8., 9., 10., 11.], 1254 [ 12., 13., 14., 15.]]) 1255 1256 1257 """, 1258 ] 1259 # skip local-file depending first example: 1260 examples = examples[1:] 1261 1262 #ipython_directive.DEBUG = True # dbg 1263 #options = dict(suppress=True) # dbg 1264 options = {} 1265 for example in examples: 1266 content = example.split('\n') 1267 IPythonDirective('debug', arguments=None, options=options, 1268 content=content, lineno=0, 1269 content_offset=None, block_text=None, 1270 state=None, state_machine=None, 1271 ) 1272 1273 # Run test suite as a script 1274 if __name__=='__main__': 1275 if not os.path.isdir('_static'): 1276 os.mkdir('_static') 1277 test() 1278 print('All OK? Check figures in _static/') 1279