[{"data":1,"prerenderedAt":46826},["ShallowReactive",2],{"blog-post-/blogs/tui-clash":3,"all-blogs-for-navigation":292},{"id":4,"title":5,"body":6,"description":279,"extension":280,"meta":281,"navigation":91,"ogImage":283,"path":288,"seo":289,"stem":290,"__hash__":291},"content/blogs/6. 从零开始在 TUI 服务器中启动 clash.md","从零开始在 TUI 服务器中启动 clash",{"type":7,"value":8,"toc":269},"minimark",[9,13,17,20,23,26,29,34,45,53,122,125,128,135,139,160,163,166,181,184,187,230,236,246,249,265],[10,11,12],"h2",{"id":12},"背景",[14,15,16],"p",{},"相信国内的朋友在用自己实验室服务器的时候总会遇到各种网络问题，比如从 GitHub 上 clone 不下来代码，push 不上去代码，下载模型，数据集非常缓慢等。网络问题已经成为大家科研路上的一个重要绊脚石！！",[14,18,19],{},"为了帮助更多的人解决这个问题，我想分享一下我的经验。",[10,21,22],{"id":22},"准备",[14,24,25],{},"这篇文章假设你已经学会了在自己电脑上使用任意一款有 GUI 页面的 Clash 进行翻墙，你需要有一个能正常工作的 Clash，正常运行的机场，并在自己的 Clash 里成功使用这个机场翻墙。",[14,27,28],{},"同时假设你没有服务器的 sudo 权限，服务器使用 Linux 系统，CPU 为 x86_64（amd64）版本。",[30,31,33],"h3",{"id":32},"下载-mihomo-内核","下载 mihomo 内核",[14,35,36,37,44],{},"从这里 ",[38,39,43],"a",{"href":40,"rel":41},"https://github.com/MetaCubeX/mihomo/releases",[42],"nofollow","MetaCubeX/mihomo"," 下载最新的 mihomo 内核。",[14,46,47,48,52],{},"从中找到 ",[49,50,51],"code",{},"mihomo-linux-amd64-compatible-v*.gz","（* 表示任意版本），然后下载，复制到你的服务器上，解压并重命名为 clash，指令如下：",[54,55,60],"pre",{"className":56,"code":57,"language":58,"meta":59,"style":59},"language-bash shiki shiki-themes github-light dracula","gzip -d mihomo-linux-amd64-compatible-v*.gz\n\nmv mihomo-linux-amd64-compatible-v* clash\n\nchmod +x clash\n","bash","",[49,61,62,86,93,106,111],{"__ignoreMap":59},[63,64,67,71,75,79,83],"span",{"class":65,"line":66},"line",1,[63,68,70],{"class":69},"sCLZk","gzip",[63,72,74],{"class":73},"soDru"," -d",[63,76,78],{"class":77},"sEzAm"," mihomo-linux-amd64-compatible-v",[63,80,82],{"class":81},"sJti5","*",[63,84,85],{"class":77},".gz\n",[63,87,89],{"class":65,"line":88},2,[63,90,92],{"emptyLinePlaceholder":91},true,"\n",[63,94,96,99,101,103],{"class":65,"line":95},3,[63,97,98],{"class":69},"mv",[63,100,78],{"class":77},[63,102,82],{"class":81},[63,104,105],{"class":77}," clash\n",[63,107,109],{"class":65,"line":108},4,[63,110,92],{"emptyLinePlaceholder":91},[63,112,114,117,120],{"class":65,"line":113},5,[63,115,116],{"class":69},"chmod",[63,118,119],{"class":77}," +x",[63,121,105],{"class":77},[14,123,124],{},"我个人建议吧解压出的 clash 单独放置一个文件夹。",[30,126,127],{"id":127},"准备机场配置",[14,129,130,131,134],{},"启动你的电脑中的 clash，点击订阅，右键你当前激活的订阅，点击编辑文件，然后将出现的 yaml 文件内容全部复制下来，在 clash 同目录下创建一个 ",[49,132,133],{},"config.yaml"," 文件，并将刚才复制的内容全部粘贴进去。",[30,136,138],{"id":137},"geox-url-准备","geox-url 准备",[14,140,141,142,147,148,153,154,159],{},"分别下载 ",[38,143,146],{"href":144,"rel":145},"https://fastly.jsdelivr.net/gh/MetaCubeX/meta-rules-dat@release/geoip.dat",[42],"geoip","，",[38,149,152],{"href":150,"rel":151},"https://fastly.jsdelivr.net/gh/MetaCubeX/meta-rules-dat@release/geosite.dat",[42],"geosite"," 和 ",[38,155,158],{"href":156,"rel":157},"https://fastly.jsdelivr.net/gh/MetaCubeX/meta-rules-dat@release/geoip.metadb",[42],"mmdb"," 到同级文件夹。",[10,161,162],{"id":162},"启动",[14,164,165],{},"执行下述指令，你的 clash 就启动成功了，从启动日志上可以看到你的监听端口，外部控制端口等信息。",[54,167,169],{"className":56,"code":168,"language":58,"meta":59,"style":59},"./clash -d ./\n",[49,170,171],{"__ignoreMap":59},[63,172,173,176,178],{"class":65,"line":66},[63,174,175],{"class":69},"./clash",[63,177,74],{"class":73},[63,179,180],{"class":77}," ./\n",[10,182,183],{"id":183},"设置系统代理",[14,185,186],{},"在一个终端中启动 clash 后，在需要访问境外服务的终端上设置系统代理，指令如下：",[54,188,190],{"className":56,"code":189,"language":58,"meta":59,"style":59},"export https_proxy=\"http://127.0.0.1:7890\"\nexport http_proxy=\"http://127.0.0.1:7890\"\n",[49,191,192,215],{"__ignoreMap":59},[63,193,194,198,202,205,209,212],{"class":65,"line":66},[63,195,197],{"class":196},"szJfE","export",[63,199,201],{"class":200},"scEb-"," https_proxy",[63,203,204],{"class":196},"=",[63,206,208],{"class":207},"sMWOi","\"",[63,210,211],{"class":77},"http://127.0.0.1:7890",[63,213,214],{"class":207},"\"\n",[63,216,217,219,222,224,226,228],{"class":65,"line":88},[63,218,197],{"class":196},[63,220,221],{"class":200}," http_proxy",[63,223,204],{"class":196},[63,225,208],{"class":207},[63,227,211],{"class":77},[63,229,214],{"class":207},[14,231,232,233,235],{},"注意这里没有输错，https_proxy 对应的就是 ",[49,234,211],{},"。",[14,237,238,239,241,242,245],{},"这里的端口可能会根据 ",[49,240,133],{}," 文件的不同而有所变化，注意日志中的 ",[49,243,244],{},"mixed_port","，实际以这个为准。",[14,247,248],{},"到这里配置就结束了，可以链接一下谷歌自行测试联通性。",[54,250,252],{"className":56,"code":251,"language":58,"meta":59,"style":59},"curl -v www.google.com\n",[49,253,254],{"__ignoreMap":59},[63,255,256,259,262],{"class":65,"line":66},[63,257,258],{"class":69},"curl",[63,260,261],{"class":73}," -v",[63,263,264],{"class":77}," www.google.com\n",[266,267,268],"style",{},"html pre.shiki code .sCLZk, html code.shiki .sCLZk{--shiki-default:#6F42C1;--shiki-dark:#50FA7B}html pre.shiki code .soDru, html code.shiki .soDru{--shiki-default:#005CC5;--shiki-dark:#BD93F9}html pre.shiki code .sEzAm, html code.shiki .sEzAm{--shiki-default:#032F62;--shiki-dark:#F1FA8C}html pre.shiki code .sJti5, html code.shiki .sJti5{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#BD93F9;--shiki-dark-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .szJfE, html code.shiki .szJfE{--shiki-default:#D73A49;--shiki-dark:#FF79C6}html pre.shiki code .scEb-, html code.shiki .scEb-{--shiki-default:#24292E;--shiki-dark:#BD93F9}html pre.shiki code .sMWOi, html code.shiki .sMWOi{--shiki-default:#032F62;--shiki-dark:#E9F284}",{"title":59,"searchDepth":88,"depth":88,"links":270},[271,272,277,278],{"id":12,"depth":88,"text":12},{"id":22,"depth":88,"text":22,"children":273},[274,275,276],{"id":32,"depth":95,"text":33},{"id":127,"depth":95,"text":127},{"id":137,"depth":95,"text":138},{"id":162,"depth":88,"text":162},{"id":183,"depth":88,"text":183},"探究如何在纯 TUI 服务器上启动 clash，解决中国境内的网络问题","md",{"date":282,"image":283,"alt":5,"tags":284,"published":91},"1 May 2026","/blogs-img/blog4.jpg",[285,286,287],"Linux","折腾日记","实用小技能","/blogs/tui-clash",{"title":5,"description":279},"blogs/6. 从零开始在 TUI 服务器中启动 clash","ZmQG4yZmy47Hg_DjZMPeOUpHQPdH-KV2mr3UoqfDXkc",[293,12746,45448,46406,46581,46646],{"id":294,"title":295,"body":296,"description":12735,"extension":280,"meta":12736,"navigation":91,"ogImage":12738,"path":12742,"seo":12743,"stem":12744,"__hash__":12745},"content/blogs/1. transformer复习笔记.md","transformer 复习笔记",{"type":7,"value":297,"toc":12719},[298,300,303,317,320,327,330,333,336,595,719,1035,1450,1456,1583,2459,2466,2683,2686,2923,2934,2937,2940,2945,3277,4623,4642,4868,5327,5330,5411,5414,6009,6351,6688,7040,7313,7316,7319,7657,7660,7817,7820,7823,8072,8211,8214,8727,9125,9129,9132,9741,9745,9748,10748,10751,10754,11045,11048,11205,11221,11814,11817,12522,12525,12703,12706,12711,12716],[10,299,12],{"id":12},[14,301,302],{},"距离首次学习 Transformer 已过去一年，最近在尝试回忆它的细节时，发现绝大部分内容已经模糊，于是决定系统地复盘一遍，并把过程记录下来，方便日后查阅。",[14,304,305,306,311,312,235],{},"本文重点记录对 Transformer 的理解与思考。代码实现主要参考 ",[38,307,310],{"href":308,"rel":309},"https://zh.d2l.ai/chapter_attention-mechanisms/transformer.html",[42],"动手学深度学习","，部分理论解释参考 ",[38,313,316],{"href":314,"rel":315},"https://nlp.seas.harvard.edu/annotated-transformer/",[42],"The Annotated Transformer",[10,318,319],{"id":319},"整体框架",[14,321,322],{},[323,324],"img",{"alt":325,"src":326},"transformer架构图","transformer/transformer.png",[10,328,329],{"id":329},"注意力机制",[30,331,332],{"id":332},"注意力函数",[14,334,335],{},"常见的注意力函数有两种：加性注意力（additive attention）与缩放点积注意力（scaled dot-product attention）。",[14,337,338,339,594],{},"缩放点积注意力相比朴素的点积注意力，多了一个缩放因子 ",[63,340,343,382],{"className":341},[342],"katex",[63,344,347],{"className":345},[346],"katex-mathml",[348,349,351],"math",{"xmlns":350},"http://www.w3.org/1998/Math/MathML",[352,353,354,377],"semantics",{},[355,356,357],"mrow",{},[358,359,360,364],"mfrac",{},[361,362,363],"mn",{},"1",[365,366,367],"msqrt",{},[368,369,370,374],"msub",{},[371,372,373],"mi",{},"d",[371,375,376],{},"k",[378,379,381],"annotation",{"encoding":380},"application/x-tex","\\frac{1}{\\sqrt{d_k}}",[63,383,387],{"className":384,"ariaHidden":386},[385],"katex-html","true",[63,388,391,396],{"className":389},[390],"base",[63,392],{"className":393,"style":395},[394],"strut","height:1.3831em;vertical-align:-0.538em;",[63,397,400,405,590],{"className":398},[399],"mord",[63,401],{"className":402},[403,404],"mopen","nulldelimiter",[63,406,408],{"className":407},[358],[63,409,413,581],{"className":410},[411,412],"vlist-t","vlist-t2",[63,414,417,578],{"className":415},[416],"vlist-r",[63,418,422,552,563],{"className":419,"style":421},[420],"vlist","height:0.8451em;",[63,423,425,430],{"style":424},"top:-2.5864em;",[63,426],{"className":427,"style":429},[428],"pstrut","height:3em;",[63,431,437],{"className":432},[433,434,435,436],"sizing","reset-size6","size3","mtight",[63,438,440],{"className":439},[399,436],[63,441,444],{"className":442},[399,443,436],"sqrt",[63,445,447,543],{"className":446},[411,412],[63,448,450,540],{"className":449},[416],[63,451,454,517],{"className":452,"style":453},[420],"height:0.8622em;",[63,455,459,462],{"className":456,"style":458},[457],"svg-align","top:-3em;",[63,460],{"className":461,"style":429},[428],[63,463,466],{"className":464,"style":465},[399,436],"padding-left:0.833em;",[63,467,469,473],{"className":468},[399,436],[63,470,373],{"className":471},[399,472,436],"mathnormal",[63,474,477],{"className":475},[476],"msupsub",[63,478,480,508],{"className":479},[411,412],[63,481,483,503],{"className":482},[416],[63,484,487],{"className":485,"style":486},[420],"height:0.3448em;",[63,488,490,494],{"style":489},"top:-2.3488em;margin-left:0em;margin-right:0.0714em;",[63,491],{"className":492,"style":493},[428],"height:2.5em;",[63,495,499],{"className":496},[433,497,498,436],"reset-size3","size1",[63,500,376],{"className":501,"style":502},[399,472,436],"margin-right:0.03148em;",[63,504,507],{"className":505},[506],"vlist-s","​",[63,509,511],{"className":510},[416],[63,512,515],{"className":513,"style":514},[420],"height:0.1512em;",[63,516],{},[63,518,520,523],{"style":519},"top:-2.8222em;",[63,521],{"className":522,"style":429},[428],[63,524,528],{"className":525,"style":527},[526,436],"hide-tail","min-width:0.853em;height:1.08em;",[529,530,536],"svg",{"xmlns":531,"width":532,"height":533,"viewBox":534,"preserveAspectRatio":535},"http://www.w3.org/2000/svg","400em","1.08em","0 0 400000 1080","xMinYMin slice",[537,538],"path",{"d":539},"M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z",[63,541,507],{"className":542},[506],[63,544,546],{"className":545},[416],[63,547,550],{"className":548,"style":549},[420],"height:0.1778em;",[63,551],{},[63,553,555,558],{"style":554},"top:-3.23em;",[63,556],{"className":557,"style":429},[428],[63,559],{"className":560,"style":562},[561],"frac-line","border-bottom-width:0.04em;",[63,564,566,569],{"style":565},"top:-3.394em;",[63,567],{"className":568,"style":429},[428],[63,570,572],{"className":571},[433,434,435,436],[63,573,575],{"className":574},[399,436],[63,576,363],{"className":577},[399,436],[63,579,507],{"className":580},[506],[63,582,584],{"className":583},[416],[63,585,588],{"className":586,"style":587},[420],"height:0.538em;",[63,589],{},[63,591],{"className":592},[593,404],"mclose","。加性注意力则通过一个带单隐藏层的前馈网络来计算兼容性分数。两者在理论复杂度上接近，但点积注意力可以直接复用高度优化的矩阵乘法实现，因此在实际中更快、也更省显存。",[14,596,597,598,628,629,658,659,688,689,235],{},"为了贴近实际工程的批处理方式，下面以 minibatch 的形式描述：给定 ",[63,599,601,615],{"className":600},[342],[63,602,604],{"className":603},[346],[348,605,606],{"xmlns":350},[352,607,608,613],{},[355,609,610],{},[371,611,612],{},"n",[378,614,612],{"encoding":380},[63,616,618],{"className":617,"ariaHidden":386},[385],[63,619,621,625],{"className":620},[390],[63,622],{"className":623,"style":624},[394],"height:0.4306em;",[63,626,612],{"className":627},[399,472]," 个查询和 ",[63,630,632,646],{"className":631},[342],[63,633,635],{"className":634},[346],[348,636,637],{"xmlns":350},[352,638,639,644],{},[355,640,641],{},[371,642,643],{},"m",[378,645,643],{"encoding":380},[63,647,649],{"className":648,"ariaHidden":386},[385],[63,650,652,655],{"className":651},[390],[63,653],{"className":654,"style":624},[394],[63,656,643],{"className":657},[399,472]," 个键-值对，查询与键的维度均为 ",[63,660,662,675],{"className":661},[342],[63,663,665],{"className":664},[346],[348,666,667],{"xmlns":350},[352,668,669,673],{},[355,670,671],{},[371,672,373],{},[378,674,373],{"encoding":380},[63,676,678],{"className":677,"ariaHidden":386},[385],[63,679,681,685],{"className":680},[390],[63,682],{"className":683,"style":684},[394],"height:0.6944em;",[63,686,373],{"className":687},[399,472],"，值的维度为 ",[63,690,692,706],{"className":691},[342],[63,693,695],{"className":694},[346],[348,696,697],{"xmlns":350},[352,698,699,704],{},[355,700,701],{},[371,702,703],{},"v",[378,705,703],{"encoding":380},[63,707,709],{"className":708,"ariaHidden":386},[385],[63,710,712,715],{"className":711},[390],[63,713],{"className":714,"style":624},[394],[63,716,703],{"className":717,"style":718},[399,472],"margin-right:0.03588em;",[14,720,721,722,834,835,934,935,1034],{},"查询 ",[63,723,725,760],{"className":724},[342],[63,726,728],{"className":727},[346],[348,729,730],{"xmlns":350},[352,731,732,757],{},[355,733,734,737,741],{},[371,735,736],{},"Q",[738,739,740],"mo",{},"∈",[742,743,744,748],"msup",{},[371,745,747],{"mathvariant":746},"double-struck","R",[355,749,750,752,755],{},[371,751,612],{},[738,753,754],{},"×",[371,756,373],{},[378,758,759],{"encoding":380},"Q \\in \\mathbb{R}^{n \\times d}",[63,761,763,785],{"className":762,"ariaHidden":386},[385],[63,764,766,770,773,778,782],{"className":765},[390],[63,767],{"className":768,"style":769},[394],"height:0.8778em;vertical-align:-0.1944em;",[63,771,736],{"className":772},[399,472],[63,774],{"className":775,"style":777},[776],"mspace","margin-right:0.2778em;",[63,779,740],{"className":780},[781],"mrel",[63,783],{"className":784,"style":777},[776],[63,786,788,792],{"className":787},[390],[63,789],{"className":790,"style":791},[394],"height:0.8491em;",[63,793,795,799],{"className":794},[399],[63,796,747],{"className":797},[399,798],"mathbb",[63,800,802],{"className":801},[476],[63,803,805],{"className":804},[411],[63,806,808],{"className":807},[416],[63,809,811],{"className":810,"style":791},[420],[63,812,814,818],{"style":813},"top:-3.063em;margin-right:0.05em;",[63,815],{"className":816,"style":817},[428],"height:2.7em;",[63,819,821],{"className":820},[433,434,435,436],[63,822,824,827,831],{"className":823},[399,436],[63,825,612],{"className":826},[399,472,436],[63,828,754],{"className":829},[830,436],"mbin",[63,832,373],{"className":833},[399,472,436],"、键 ",[63,836,838,867],{"className":837},[342],[63,839,841],{"className":840},[346],[348,842,843],{"xmlns":350},[352,844,845,864],{},[355,846,847,850,852],{},[371,848,849],{},"K",[738,851,740],{},[742,853,854,856],{},[371,855,747],{"mathvariant":746},[355,857,858,860,862],{},[371,859,643],{},[738,861,754],{},[371,863,373],{},[378,865,866],{"encoding":380},"K \\in \\mathbb{R}^{m \\times d}",[63,868,870,890],{"className":869,"ariaHidden":386},[385],[63,871,873,877,881,884,887],{"className":872},[390],[63,874],{"className":875,"style":876},[394],"height:0.7224em;vertical-align:-0.0391em;",[63,878,849],{"className":879,"style":880},[399,472],"margin-right:0.07153em;",[63,882],{"className":883,"style":777},[776],[63,885,740],{"className":886},[781],[63,888],{"className":889,"style":777},[776],[63,891,893,896],{"className":892},[390],[63,894],{"className":895,"style":791},[394],[63,897,899,902],{"className":898},[399],[63,900,747],{"className":901},[399,798],[63,903,905],{"className":904},[476],[63,906,908],{"className":907},[411],[63,909,911],{"className":910},[416],[63,912,914],{"className":913,"style":791},[420],[63,915,916,919],{"style":813},[63,917],{"className":918,"style":817},[428],[63,920,922],{"className":921},[433,434,435,436],[63,923,925,928,931],{"className":924},[399,436],[63,926,643],{"className":927},[399,472,436],[63,929,754],{"className":930},[830,436],[63,932,373],{"className":933},[399,472,436]," 与值 ",[63,936,938,967],{"className":937},[342],[63,939,941],{"className":940},[346],[348,942,943],{"xmlns":350},[352,944,945,964],{},[355,946,947,950,952],{},[371,948,949],{},"V",[738,951,740],{},[742,953,954,956],{},[371,955,747],{"mathvariant":746},[355,957,958,960,962],{},[371,959,643],{},[738,961,754],{},[371,963,703],{},[378,965,966],{"encoding":380},"V \\in \\mathbb{R}^{m \\times v}",[63,968,970,989],{"className":969,"ariaHidden":386},[385],[63,971,973,976,980,983,986],{"className":972},[390],[63,974],{"className":975,"style":876},[394],[63,977,949],{"className":978,"style":979},[399,472],"margin-right:0.22222em;",[63,981],{"className":982,"style":777},[776],[63,984,740],{"className":985},[781],[63,987],{"className":988,"style":777},[776],[63,990,992,996],{"className":991},[390],[63,993],{"className":994,"style":995},[394],"height:0.7713em;",[63,997,999,1002],{"className":998},[399],[63,1000,747],{"className":1001},[399,798],[63,1003,1005],{"className":1004},[476],[63,1006,1008],{"className":1007},[411],[63,1009,1011],{"className":1010},[416],[63,1012,1014],{"className":1013,"style":995},[420],[63,1015,1016,1019],{"style":813},[63,1017],{"className":1018,"style":817},[428],[63,1020,1022],{"className":1021},[433,434,435,436],[63,1023,1025,1028,1031],{"className":1024},[399,436],[63,1026,643],{"className":1027},[399,472,436],[63,1029,754],{"className":1030},[830,436],[63,1032,703],{"className":1033,"style":718},[399,472,436]," 的缩放点积注意力定义为：",[14,1036,1037],{},[63,1038,1040,1146],{"className":1039},[342],[63,1041,1043],{"className":1042},[346],[348,1044,1045],{"xmlns":350},[352,1046,1047,1143],{},[355,1048,1049,1075,1079,1081,1084,1086,1088,1090,1093,1095,1098,1101,1124,1126,1128,1140],{},[355,1050,1051,1055,1058,1060,1063,1065,1067,1070,1073],{},[371,1052,1054],{"mathvariant":1053},"normal","A",[371,1056,1057],{"mathvariant":1053},"t",[371,1059,1057],{"mathvariant":1053},[371,1061,1062],{"mathvariant":1053},"e",[371,1064,612],{"mathvariant":1053},[371,1066,1057],{"mathvariant":1053},[371,1068,1069],{"mathvariant":1053},"i",[371,1071,1072],{"mathvariant":1053},"o",[371,1074,612],{"mathvariant":1053},[738,1076,1078],{"stretchy":1077},"false","(",[371,1080,736],{},[738,1082,1083],{"separator":386},",",[371,1085,849],{},[738,1087,1083],{"separator":386},[371,1089,949],{},[738,1091,1092],{"stretchy":1077},")",[738,1094,204],{},[371,1096,1097],{"mathvariant":1053},"softmax",[738,1099,1100],{},"⁡",[355,1102,1103,1105,1122],{},[738,1104,1078],{"fence":386},[358,1106,1107,1118],{},[355,1108,1109,1111],{},[371,1110,736],{},[742,1112,1113,1115],{},[371,1114,849],{},[371,1116,1117],{"mathvariant":1053},"⊤",[365,1119,1120],{},[371,1121,373],{},[738,1123,1092],{"fence":386},[371,1125,949],{},[738,1127,740],{},[742,1129,1130,1132],{},[371,1131,747],{"mathvariant":746},[355,1133,1134,1136,1138],{},[371,1135,612],{},[738,1137,754],{},[371,1139,703],{},[371,1141,1142],{"mathvariant":1053},".",[378,1144,1145],{"encoding":380}," \\mathrm{Attention}(Q,K,V) = \\operatorname{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d}}\\right)V \\in \\mathbb{R}^{n \\times v}.",[63,1147,1149,1202,1403],{"className":1148,"ariaHidden":386},[385],[63,1150,1152,1156,1164,1167,1170,1174,1178,1181,1184,1187,1190,1193,1196,1199],{"className":1151},[390],[63,1153],{"className":1154,"style":1155},[394],"height:1em;vertical-align:-0.25em;",[63,1157,1159],{"className":1158},[399],[63,1160,1163],{"className":1161},[399,1162],"mathrm","Attention",[63,1165,1078],{"className":1166},[403],[63,1168,736],{"className":1169},[399,472],[63,1171,1083],{"className":1172},[1173],"mpunct",[63,1175],{"className":1176,"style":1177},[776],"margin-right:0.1667em;",[63,1179,849],{"className":1180,"style":880},[399,472],[63,1182,1083],{"className":1183},[1173],[63,1185],{"className":1186,"style":1177},[776],[63,1188,949],{"className":1189,"style":979},[399,472],[63,1191,1092],{"className":1192},[593],[63,1194],{"className":1195,"style":777},[776],[63,1197,204],{"className":1198},[781],[63,1200],{"className":1201,"style":777},[776],[63,1203,1205,1209,1216,1219,1388,1391,1394,1397,1400],{"className":1204},[390],[63,1206],{"className":1207,"style":1208},[394],"height:1.8em;vertical-align:-0.65em;",[63,1210,1213],{"className":1211},[1212],"mop",[63,1214,1097],{"className":1215},[399,1162],[63,1217],{"className":1218,"style":1177},[776],[63,1220,1223,1233,1382],{"className":1221},[1222],"minner",[63,1224,1228],{"className":1225,"style":1227},[403,1226],"delimcenter","top:0em;",[63,1229,1078],{"className":1230},[1231,1232],"delimsizing","size2",[63,1234,1236,1239,1379],{"className":1235},[399],[63,1237],{"className":1238},[403,404],[63,1240,1242],{"className":1241},[358],[63,1243,1245,1371],{"className":1244},[411,412],[63,1246,1248,1368],{"className":1247},[416],[63,1249,1252,1314,1322],{"className":1250,"style":1251},[420],"height:1.095em;",[63,1253,1255,1258],{"style":1254},"top:-2.5335em;",[63,1256],{"className":1257,"style":429},[428],[63,1259,1261],{"className":1260},[433,434,435,436],[63,1262,1264],{"className":1263},[399,436],[63,1265,1267],{"className":1266},[399,443,436],[63,1268,1270,1305],{"className":1269},[411,412],[63,1271,1273,1302],{"className":1272},[416],[63,1274,1277,1289],{"className":1275,"style":1276},[420],"height:0.9378em;",[63,1278,1280,1283],{"className":1279,"style":458},[457],[63,1281],{"className":1282,"style":429},[428],[63,1284,1286],{"className":1285,"style":465},[399,436],[63,1287,373],{"className":1288},[399,472,436],[63,1290,1292,1295],{"style":1291},"top:-2.8978em;",[63,1293],{"className":1294,"style":429},[428],[63,1296,1298],{"className":1297,"style":527},[526,436],[529,1299,1300],{"xmlns":531,"width":532,"height":533,"viewBox":534,"preserveAspectRatio":535},[537,1301],{"d":539},[63,1303,507],{"className":1304},[506],[63,1306,1308],{"className":1307},[416],[63,1309,1312],{"className":1310,"style":1311},[420],"height:0.1022em;",[63,1313],{},[63,1315,1316,1319],{"style":554},[63,1317],{"className":1318,"style":429},[428],[63,1320],{"className":1321,"style":562},[561],[63,1323,1325,1328],{"style":1324},"top:-3.4461em;",[63,1326],{"className":1327,"style":429},[428],[63,1329,1331],{"className":1330},[433,434,435,436],[63,1332,1334,1337],{"className":1333},[399,436],[63,1335,736],{"className":1336},[399,472,436],[63,1338,1340,1343],{"className":1339},[399,436],[63,1341,849],{"className":1342,"style":880},[399,472,436],[63,1344,1346],{"className":1345},[476],[63,1347,1349],{"className":1348},[411],[63,1350,1352],{"className":1351},[416],[63,1353,1356],{"className":1354,"style":1355},[420],"height:0.927em;",[63,1357,1359,1362],{"style":1358},"top:-2.931em;margin-right:0.0714em;",[63,1360],{"className":1361,"style":493},[428],[63,1363,1365],{"className":1364},[433,497,498,436],[63,1366,1117],{"className":1367},[399,436],[63,1369,507],{"className":1370},[506],[63,1372,1374],{"className":1373},[416],[63,1375,1377],{"className":1376,"style":587},[420],[63,1378],{},[63,1380],{"className":1381},[593,404],[63,1383,1385],{"className":1384,"style":1227},[593,1226],[63,1386,1092],{"className":1387},[1231,1232],[63,1389],{"className":1390,"style":1177},[776],[63,1392,949],{"className":1393,"style":979},[399,472],[63,1395],{"className":1396,"style":777},[776],[63,1398,740],{"className":1399},[781],[63,1401],{"className":1402,"style":777},[776],[63,1404,1406,1409,1447],{"className":1405},[390],[63,1407],{"className":1408,"style":995},[394],[63,1410,1412,1415],{"className":1411},[399],[63,1413,747],{"className":1414},[399,798],[63,1416,1418],{"className":1417},[476],[63,1419,1421],{"className":1420},[411],[63,1422,1424],{"className":1423},[416],[63,1425,1427],{"className":1426,"style":995},[420],[63,1428,1429,1432],{"style":813},[63,1430],{"className":1431,"style":817},[428],[63,1433,1435],{"className":1434},[433,434,435,436],[63,1436,1438,1441,1444],{"className":1437},[399,436],[63,1439,612],{"className":1440},[399,472,436],[63,1442,754],{"className":1443},[830,436],[63,1445,703],{"className":1446,"style":718},[399,472,436],[63,1448,1142],{"className":1449},[399],[14,1451,1452],{},[323,1453],{"alt":1454,"src":1455},"注意力","transformer/attention.png",[14,1457,1458,1459,1582],{},"具体过程是：先用查询与所有键做点积，再除以 ",[63,1460,1462,1482],{"className":1461},[342],[63,1463,1465],{"className":1464},[346],[348,1466,1467],{"xmlns":350},[352,1468,1469,1479],{},[355,1470,1471],{},[365,1472,1473],{},[368,1474,1475,1477],{},[371,1476,373],{},[371,1478,376],{},[378,1480,1481],{"encoding":380},"\\sqrt{d_k}",[63,1483,1485],{"className":1484,"ariaHidden":386},[385],[63,1486,1488,1492],{"className":1487},[390],[63,1489],{"className":1490,"style":1491},[394],"height:1.04em;vertical-align:-0.1828em;",[63,1493,1495],{"className":1494},[399,443],[63,1496,1498,1573],{"className":1497},[411,412],[63,1499,1501,1570],{"className":1500},[416],[63,1502,1505,1557],{"className":1503,"style":1504},[420],"height:0.8572em;",[63,1506,1508,1511],{"className":1507,"style":458},[457],[63,1509],{"className":1510,"style":429},[428],[63,1512,1514],{"className":1513,"style":465},[399],[63,1515,1517,1520],{"className":1516},[399],[63,1518,373],{"className":1519},[399,472],[63,1521,1523],{"className":1522},[476],[63,1524,1526,1548],{"className":1525},[411,412],[63,1527,1529,1545],{"className":1528},[416],[63,1530,1533],{"className":1531,"style":1532},[420],"height:0.3361em;",[63,1534,1536,1539],{"style":1535},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[63,1537],{"className":1538,"style":817},[428],[63,1540,1542],{"className":1541},[433,434,435,436],[63,1543,376],{"className":1544,"style":502},[399,472,436],[63,1546,507],{"className":1547},[506],[63,1549,1551],{"className":1550},[416],[63,1552,1555],{"className":1553,"style":1554},[420],"height:0.15em;",[63,1556],{},[63,1558,1560,1563],{"style":1559},"top:-2.8172em;",[63,1561],{"className":1562,"style":429},[428],[63,1564,1566],{"className":1565,"style":527},[526],[529,1567,1568],{"xmlns":531,"width":532,"height":533,"viewBox":534,"preserveAspectRatio":535},[537,1569],{"d":539},[63,1571,507],{"className":1572},[506],[63,1574,1576],{"className":1575},[416],[63,1577,1580],{"className":1578,"style":1579},[420],"height:0.1828em;",[63,1581],{}," 进行缩放，然后通过 softmax 得到值上的权重分布，最后对值做加权求和。",[14,1584,1585,1586,1657,1658,1727,1728,1758,1759,1787,1788,1818,1819,1847,1848,2145,2146,2174,2175,2244,2245,2273,2274,2458],{},"那么，为什么需要这个缩放因子呢？对较小的 ",[63,1587,1589,1607],{"className":1588},[342],[63,1590,1592],{"className":1591},[346],[348,1593,1594],{"xmlns":350},[352,1595,1596,1604],{},[355,1597,1598],{},[368,1599,1600,1602],{},[371,1601,373],{},[371,1603,376],{},[378,1605,1606],{"encoding":380},"d_k",[63,1608,1610],{"className":1609,"ariaHidden":386},[385],[63,1611,1613,1617],{"className":1612},[390],[63,1614],{"className":1615,"style":1616},[394],"height:0.8444em;vertical-align:-0.15em;",[63,1618,1620,1623],{"className":1619},[399],[63,1621,373],{"className":1622},[399,472],[63,1624,1626],{"className":1625},[476],[63,1627,1629,1649],{"className":1628},[411,412],[63,1630,1632,1646],{"className":1631},[416],[63,1633,1635],{"className":1634,"style":1532},[420],[63,1636,1637,1640],{"style":1535},[63,1638],{"className":1639,"style":817},[428],[63,1641,1643],{"className":1642},[433,434,435,436],[63,1644,376],{"className":1645,"style":502},[399,472,436],[63,1647,507],{"className":1648},[506],[63,1650,1652],{"className":1651},[416],[63,1653,1655],{"className":1654,"style":1554},[420],[63,1656],{},"，加性注意力与点积注意力表现接近；但当 ",[63,1659,1661,1678],{"className":1660},[342],[63,1662,1664],{"className":1663},[346],[348,1665,1666],{"xmlns":350},[352,1667,1668,1676],{},[355,1669,1670],{},[368,1671,1672,1674],{},[371,1673,373],{},[371,1675,376],{},[378,1677,1606],{"encoding":380},[63,1679,1681],{"className":1680,"ariaHidden":386},[385],[63,1682,1684,1687],{"className":1683},[390],[63,1685],{"className":1686,"style":1616},[394],[63,1688,1690,1693],{"className":1689},[399],[63,1691,373],{"className":1692},[399,472],[63,1694,1696],{"className":1695},[476],[63,1697,1699,1719],{"className":1698},[411,412],[63,1700,1702,1716],{"className":1701},[416],[63,1703,1705],{"className":1704,"style":1532},[420],[63,1706,1707,1710],{"style":1535},[63,1708],{"className":1709,"style":817},[428],[63,1711,1713],{"className":1712},[433,434,435,436],[63,1714,376],{"className":1715,"style":502},[399,472,436],[63,1717,507],{"className":1718},[506],[63,1720,1722],{"className":1721},[416],[63,1723,1725],{"className":1724,"style":1554},[420],[63,1726],{}," 较大时，加性注意力反而更优。原因在于：当维度很高时，点积的数值会被放大，导致 softmax 的输出几乎集中在一个位置上，梯度也随之消失。可以做一个简化的方差分析：假设查询 ",[63,1729,1731,1745],{"className":1730},[342],[63,1732,1734],{"className":1733},[346],[348,1735,1736],{"xmlns":350},[352,1737,1738,1743],{},[355,1739,1740],{},[371,1741,1742],{},"q",[378,1744,1742],{"encoding":380},[63,1746,1748],{"className":1747,"ariaHidden":386},[385],[63,1749,1751,1755],{"className":1750},[390],[63,1752],{"className":1753,"style":1754},[394],"height:0.625em;vertical-align:-0.1944em;",[63,1756,1742],{"className":1757,"style":718},[399,472]," 与键 ",[63,1760,1762,1775],{"className":1761},[342],[63,1763,1765],{"className":1764},[346],[348,1766,1767],{"xmlns":350},[352,1768,1769,1773],{},[355,1770,1771],{},[371,1772,376],{},[378,1774,376],{"encoding":380},[63,1776,1778],{"className":1777,"ariaHidden":386},[385],[63,1779,1781,1784],{"className":1780},[390],[63,1782],{"className":1783,"style":684},[394],[63,1785,376],{"className":1786,"style":502},[399,472]," 的每个分量都是均值 ",[63,1789,1791,1805],{"className":1790},[342],[63,1792,1794],{"className":1793},[346],[348,1795,1796],{"xmlns":350},[352,1797,1798,1803],{},[355,1799,1800],{},[361,1801,1802],{},"0",[378,1804,1802],{"encoding":380},[63,1806,1808],{"className":1807,"ariaHidden":386},[385],[63,1809,1811,1815],{"className":1810},[390],[63,1812],{"className":1813,"style":1814},[394],"height:0.6444em;",[63,1816,1802],{"className":1817},[399],"、方差 ",[63,1820,1822,1835],{"className":1821},[342],[63,1823,1825],{"className":1824},[346],[348,1826,1827],{"xmlns":350},[352,1828,1829,1833],{},[355,1830,1831],{},[361,1832,363],{},[378,1834,363],{"encoding":380},[63,1836,1838],{"className":1837,"ariaHidden":386},[385],[63,1839,1841,1844],{"className":1840},[390],[63,1842],{"className":1843,"style":1814},[394],[63,1845,363],{"className":1846},[399]," 的独立随机变量，则它们的点积 ",[63,1849,1851,1904],{"className":1850},[342],[63,1852,1854],{"className":1853},[346],[348,1855,1856],{"xmlns":350},[352,1857,1858,1901],{},[355,1859,1860,1862,1865,1867,1869,1889,1895],{},[371,1861,1742],{},[738,1863,1864],{},"⋅",[371,1866,376],{},[738,1868,204],{},[1870,1871,1872,1875,1883],"msubsup",{},[738,1873,1874],{},"∑",[355,1876,1877,1879,1881],{},[371,1878,1069],{},[738,1880,204],{},[361,1882,363],{},[368,1884,1885,1887],{},[371,1886,373],{},[371,1888,376],{},[368,1890,1891,1893],{},[371,1892,1742],{},[371,1894,1069],{},[368,1896,1897,1899],{},[371,1898,376],{},[371,1900,1069],{},[378,1902,1903],{"encoding":380},"q \\cdot k = \\sum_{i=1}^{d_k} q_i k_i",[63,1905,1907,1927,1945],{"className":1906,"ariaHidden":386},[385],[63,1908,1910,1914,1917,1921,1924],{"className":1909},[390],[63,1911],{"className":1912,"style":1913},[394],"height:0.6389em;vertical-align:-0.1944em;",[63,1915,1742],{"className":1916,"style":718},[399,472],[63,1918],{"className":1919,"style":1920},[776],"margin-right:0.2222em;",[63,1922,1864],{"className":1923},[830],[63,1925],{"className":1926,"style":1920},[776],[63,1928,1930,1933,1936,1939,1942],{"className":1929},[390],[63,1931],{"className":1932,"style":684},[394],[63,1934,376],{"className":1935,"style":502},[399,472],[63,1937],{"className":1938,"style":777},[776],[63,1940,204],{"className":1941},[781],[63,1943],{"className":1944,"style":777},[776],[63,1946,1948,1952,2059,2062,2104],{"className":1947},[390],[63,1949],{"className":1950,"style":1951},[394],"height:1.2887em;vertical-align:-0.2997em;",[63,1953,1955,1961],{"className":1954},[1212],[63,1956,1874],{"className":1957,"style":1960},[1212,1958,1959],"op-symbol","small-op","position:relative;top:0em;",[63,1962,1964],{"className":1963},[476],[63,1965,1967,2050],{"className":1966},[411,412],[63,1968,1970,2047],{"className":1969},[416],[63,1971,1974,1995],{"className":1972,"style":1973},[420],"height:0.989em;",[63,1975,1977,1980],{"style":1976},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[63,1978],{"className":1979,"style":817},[428],[63,1981,1983],{"className":1982},[433,434,435,436],[63,1984,1986,1989,1992],{"className":1985},[399,436],[63,1987,1069],{"className":1988},[399,472,436],[63,1990,204],{"className":1991},[781,436],[63,1993,363],{"className":1994},[399,436],[63,1996,1998,2001],{"style":1997},"top:-3.2029em;margin-right:0.05em;",[63,1999],{"className":2000,"style":817},[428],[63,2002,2004],{"className":2003},[433,434,435,436],[63,2005,2007],{"className":2006},[399,436],[63,2008,2010,2013],{"className":2009},[399,436],[63,2011,373],{"className":2012},[399,472,436],[63,2014,2016],{"className":2015},[476],[63,2017,2019,2039],{"className":2018},[411,412],[63,2020,2022,2036],{"className":2021},[416],[63,2023,2025],{"className":2024,"style":486},[420],[63,2026,2027,2030],{"style":489},[63,2028],{"className":2029,"style":493},[428],[63,2031,2033],{"className":2032},[433,497,498,436],[63,2034,376],{"className":2035,"style":502},[399,472,436],[63,2037,507],{"className":2038},[506],[63,2040,2042],{"className":2041},[416],[63,2043,2045],{"className":2044,"style":514},[420],[63,2046],{},[63,2048,507],{"className":2049},[506],[63,2051,2053],{"className":2052},[416],[63,2054,2057],{"className":2055,"style":2056},[420],"height:0.2997em;",[63,2058],{},[63,2060],{"className":2061,"style":1177},[776],[63,2063,2065,2068],{"className":2064},[399],[63,2066,1742],{"className":2067,"style":718},[399,472],[63,2069,2071],{"className":2070},[476],[63,2072,2074,2096],{"className":2073},[411,412],[63,2075,2077,2093],{"className":2076},[416],[63,2078,2081],{"className":2079,"style":2080},[420],"height:0.3117em;",[63,2082,2084,2087],{"style":2083},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[63,2085],{"className":2086,"style":817},[428],[63,2088,2090],{"className":2089},[433,434,435,436],[63,2091,1069],{"className":2092},[399,472,436],[63,2094,507],{"className":2095},[506],[63,2097,2099],{"className":2098},[416],[63,2100,2102],{"className":2101,"style":1554},[420],[63,2103],{},[63,2105,2107,2110],{"className":2106},[399],[63,2108,376],{"className":2109,"style":502},[399,472],[63,2111,2113],{"className":2112},[476],[63,2114,2116,2137],{"className":2115},[411,412],[63,2117,2119,2134],{"className":2118},[416],[63,2120,2122],{"className":2121,"style":2080},[420],[63,2123,2125,2128],{"style":2124},"top:-2.55em;margin-left:-0.0315em;margin-right:0.05em;",[63,2126],{"className":2127,"style":817},[428],[63,2129,2131],{"className":2130},[433,434,435,436],[63,2132,1069],{"className":2133},[399,472,436],[63,2135,507],{"className":2136},[506],[63,2138,2140],{"className":2139},[416],[63,2141,2143],{"className":2142,"style":1554},[420],[63,2144],{}," 的均值为 ",[63,2147,2149,2162],{"className":2148},[342],[63,2150,2152],{"className":2151},[346],[348,2153,2154],{"xmlns":350},[352,2155,2156,2160],{},[355,2157,2158],{},[361,2159,1802],{},[378,2161,1802],{"encoding":380},[63,2163,2165],{"className":2164,"ariaHidden":386},[385],[63,2166,2168,2171],{"className":2167},[390],[63,2169],{"className":2170,"style":1814},[394],[63,2172,1802],{"className":2173},[399],"、方差为 ",[63,2176,2178,2195],{"className":2177},[342],[63,2179,2181],{"className":2180},[346],[348,2182,2183],{"xmlns":350},[352,2184,2185,2193],{},[355,2186,2187],{},[368,2188,2189,2191],{},[371,2190,373],{},[371,2192,376],{},[378,2194,1606],{"encoding":380},[63,2196,2198],{"className":2197,"ariaHidden":386},[385],[63,2199,2201,2204],{"className":2200},[390],[63,2202],{"className":2203,"style":1616},[394],[63,2205,2207,2210],{"className":2206},[399],[63,2208,373],{"className":2209},[399,472],[63,2211,2213],{"className":2212},[476],[63,2214,2216,2236],{"className":2215},[411,412],[63,2217,2219,2233],{"className":2218},[416],[63,2220,2222],{"className":2221,"style":1532},[420],[63,2223,2224,2227],{"style":1535},[63,2225],{"className":2226,"style":817},[428],[63,2228,2230],{"className":2229},[433,434,435,436],[63,2231,376],{"className":2232,"style":502},[399,472,436],[63,2234,507],{"className":2235},[506],[63,2237,2239],{"className":2238},[416],[63,2240,2242],{"className":2241,"style":1554},[420],[63,2243],{},"。为了把方差稳定回 ",[63,2246,2248,2261],{"className":2247},[342],[63,2249,2251],{"className":2250},[346],[348,2252,2253],{"xmlns":350},[352,2254,2255,2259],{},[355,2256,2257],{},[361,2258,363],{},[378,2260,363],{"encoding":380},[63,2262,2264],{"className":2263,"ariaHidden":386},[385],[63,2265,2267,2270],{"className":2266},[390],[63,2268],{"className":2269,"style":1814},[394],[63,2271,363],{"className":2272},[399],"，就需要乘上 ",[63,2275,2277,2300],{"className":2276},[342],[63,2278,2280],{"className":2279},[346],[348,2281,2282],{"xmlns":350},[352,2283,2284,2298],{},[355,2285,2286],{},[358,2287,2288,2290],{},[361,2289,363],{},[365,2291,2292],{},[368,2293,2294,2296],{},[371,2295,373],{},[371,2297,376],{},[378,2299,381],{"encoding":380},[63,2301,2303],{"className":2302,"ariaHidden":386},[385],[63,2304,2306,2309],{"className":2305},[390],[63,2307],{"className":2308,"style":395},[394],[63,2310,2312,2315,2455],{"className":2311},[399],[63,2313],{"className":2314},[403,404],[63,2316,2318],{"className":2317},[358],[63,2319,2321,2447],{"className":2320},[411,412],[63,2322,2324,2444],{"className":2323},[416],[63,2325,2327,2422,2430],{"className":2326,"style":421},[420],[63,2328,2329,2332],{"style":424},[63,2330],{"className":2331,"style":429},[428],[63,2333,2335],{"className":2334},[433,434,435,436],[63,2336,2338],{"className":2337},[399,436],[63,2339,2341],{"className":2340},[399,443,436],[63,2342,2344,2414],{"className":2343},[411,412],[63,2345,2347,2411],{"className":2346},[416],[63,2348,2350,2399],{"className":2349,"style":453},[420],[63,2351,2353,2356],{"className":2352,"style":458},[457],[63,2354],{"className":2355,"style":429},[428],[63,2357,2359],{"className":2358,"style":465},[399,436],[63,2360,2362,2365],{"className":2361},[399,436],[63,2363,373],{"className":2364},[399,472,436],[63,2366,2368],{"className":2367},[476],[63,2369,2371,2391],{"className":2370},[411,412],[63,2372,2374,2388],{"className":2373},[416],[63,2375,2377],{"className":2376,"style":486},[420],[63,2378,2379,2382],{"style":489},[63,2380],{"className":2381,"style":493},[428],[63,2383,2385],{"className":2384},[433,497,498,436],[63,2386,376],{"className":2387,"style":502},[399,472,436],[63,2389,507],{"className":2390},[506],[63,2392,2394],{"className":2393},[416],[63,2395,2397],{"className":2396,"style":514},[420],[63,2398],{},[63,2400,2401,2404],{"style":519},[63,2402],{"className":2403,"style":429},[428],[63,2405,2407],{"className":2406,"style":527},[526,436],[529,2408,2409],{"xmlns":531,"width":532,"height":533,"viewBox":534,"preserveAspectRatio":535},[537,2410],{"d":539},[63,2412,507],{"className":2413},[506],[63,2415,2417],{"className":2416},[416],[63,2418,2420],{"className":2419,"style":549},[420],[63,2421],{},[63,2423,2424,2427],{"style":554},[63,2425],{"className":2426,"style":429},[428],[63,2428],{"className":2429,"style":562},[561],[63,2431,2432,2435],{"style":565},[63,2433],{"className":2434,"style":429},[428],[63,2436,2438],{"className":2437},[433,434,435,436],[63,2439,2441],{"className":2440},[399,436],[63,2442,363],{"className":2443},[399,436],[63,2445,507],{"className":2446},[506],[63,2448,2450],{"className":2449},[416],[63,2451,2453],{"className":2452,"style":587},[420],[63,2454],{},[63,2456],{"className":2457},[593,404]," 这个缩放因子。",[14,2460,2461,2462,2465],{},"由于训练与推理时并非所有位置都应进入注意力汇聚（例如 padding 位置、解码器的未来位置），我们先实现一个 ",[49,2463,2464],{},"masked_softmax"," 函数，确保只有有效位置参与计算：",[54,2467,2471],{"className":2468,"code":2469,"language":2470,"meta":59,"style":59},"language-python shiki shiki-themes github-light dracula","def masked_softmax(X, valid_lens):\n    \"\"\"通过在最后一个轴上掩蔽元素来执行softmax操作\"\"\"\n    # X:3D张量，valid_lens:1D或2D张量\n    if valid_lens is None:\n        return nn.functional.softmax(X, dim=-1)\n    else:\n        shape = X.shape\n        if valid_lens.dim() == 1:\n            valid_lens = torch.repeat_interleave(valid_lens, shape[1])\n        else:\n            valid_lens = valid_lens.reshape(-1)\n        # 最后一轴上被掩蔽的元素使用一个非常大的负值替换，从而其softmax输出为0\n        X = d2l.sequence_mask(X.reshape(-1, shape[-1]), valid_lens,\n                              value=-1e6)\n        return nn.functional.softmax(X.reshape(shape), dim=-1)\n","python",[49,2472,2473,2497,2503,2509,2526,2546,2554,2565,2582,2598,2606,2623,2629,2654,2667],{"__ignoreMap":59},[63,2474,2475,2478,2481,2484,2488,2491,2494],{"class":65,"line":66},[63,2476,2477],{"class":196},"def",[63,2479,2480],{"class":69}," masked_softmax",[63,2482,1078],{"class":2483},"scbbO",[63,2485,2487],{"class":2486},"syNf4","X",[63,2489,2490],{"class":2483},", ",[63,2492,2493],{"class":2486},"valid_lens",[63,2495,2496],{"class":2483},"):\n",[63,2498,2499],{"class":65,"line":88},[63,2500,2502],{"class":2501},"seLWX","    \"\"\"通过在最后一个轴上掩蔽元素来执行softmax操作\"\"\"\n",[63,2504,2505],{"class":65,"line":95},[63,2506,2508],{"class":2507},"sfgPZ","    # X:3D张量，valid_lens:1D或2D张量\n",[63,2510,2511,2514,2517,2520,2523],{"class":65,"line":108},[63,2512,2513],{"class":196},"    if",[63,2515,2516],{"class":2483}," valid_lens ",[63,2518,2519],{"class":196},"is",[63,2521,2522],{"class":73}," None",[63,2524,2525],{"class":2483},":\n",[63,2527,2528,2531,2534,2538,2541,2543],{"class":65,"line":113},[63,2529,2530],{"class":196},"        return",[63,2532,2533],{"class":2483}," nn.functional.softmax(X, ",[63,2535,2537],{"class":2536},"sQkXh","dim",[63,2539,2540],{"class":196},"=-",[63,2542,363],{"class":73},[63,2544,2545],{"class":2483},")\n",[63,2547,2549,2552],{"class":65,"line":2548},6,[63,2550,2551],{"class":196},"    else",[63,2553,2525],{"class":2483},[63,2555,2557,2560,2562],{"class":65,"line":2556},7,[63,2558,2559],{"class":2483},"        shape ",[63,2561,204],{"class":196},[63,2563,2564],{"class":2483}," X.shape\n",[63,2566,2568,2571,2574,2577,2580],{"class":65,"line":2567},8,[63,2569,2570],{"class":196},"        if",[63,2572,2573],{"class":2483}," valid_lens.dim() ",[63,2575,2576],{"class":196},"==",[63,2578,2579],{"class":73}," 1",[63,2581,2525],{"class":2483},[63,2583,2585,2588,2590,2593,2595],{"class":65,"line":2584},9,[63,2586,2587],{"class":2483},"            valid_lens ",[63,2589,204],{"class":196},[63,2591,2592],{"class":2483}," torch.repeat_interleave(valid_lens, shape[",[63,2594,363],{"class":73},[63,2596,2597],{"class":2483},"])\n",[63,2599,2601,2604],{"class":65,"line":2600},10,[63,2602,2603],{"class":196},"        else",[63,2605,2525],{"class":2483},[63,2607,2609,2611,2613,2616,2619,2621],{"class":65,"line":2608},11,[63,2610,2587],{"class":2483},[63,2612,204],{"class":196},[63,2614,2615],{"class":2483}," valid_lens.reshape(",[63,2617,2618],{"class":196},"-",[63,2620,363],{"class":73},[63,2622,2545],{"class":2483},[63,2624,2626],{"class":65,"line":2625},12,[63,2627,2628],{"class":2507},"        # 最后一轴上被掩蔽的元素使用一个非常大的负值替换，从而其softmax输出为0\n",[63,2630,2632,2635,2637,2640,2642,2644,2647,2649,2651],{"class":65,"line":2631},13,[63,2633,2634],{"class":2483},"        X ",[63,2636,204],{"class":196},[63,2638,2639],{"class":2483}," d2l.sequence_mask(X.reshape(",[63,2641,2618],{"class":196},[63,2643,363],{"class":73},[63,2645,2646],{"class":2483},", shape[",[63,2648,2618],{"class":196},[63,2650,363],{"class":73},[63,2652,2653],{"class":2483},"]), valid_lens,\n",[63,2655,2657,2660,2662,2665],{"class":65,"line":2656},14,[63,2658,2659],{"class":2536},"                              value",[63,2661,2540],{"class":196},[63,2663,2664],{"class":73},"1e6",[63,2666,2545],{"class":2483},[63,2668,2670,2672,2675,2677,2679,2681],{"class":65,"line":2669},15,[63,2671,2530],{"class":196},[63,2673,2674],{"class":2483}," nn.functional.softmax(X.reshape(shape), ",[63,2676,2537],{"class":2536},[63,2678,2540],{"class":196},[63,2680,363],{"class":73},[63,2682,2545],{"class":2483},[14,2684,2685],{},"下面实现缩放点积注意力。",[54,2687,2689],{"className":2468,"code":2688,"language":2470,"meta":59,"style":59},"class DotProductAttention(nn.Module):\n    \"\"\"缩放点积注意力\"\"\"\n    def __init__(self, dropout, **kwargs):\n        super(DotProductAttention, self).__init__(**kwargs)\n        self.dropout = nn.Dropout(dropout)\n\n    # queries的形状：(batch_size，查询的个数，d)\n    # keys的形状：(batch_size，“键－值”对的个数，d)\n    # values的形状：(batch_size，“键－值”对的个数，值的维度)\n    # valid_lens的形状:(batch_size，)或者(batch_size，查询的个数)\n    def forward(self, queries, keys, values, valid_lens=None):\n        d = queries.shape[-1]\n        # 设置transpose_b=True为了交换keys的最后两个维度\n        scores = torch.bmm(queries, keys.transpose(1,2)) / math.sqrt(d)\n        self.attention_weights = masked_softmax(scores, valid_lens)\n        return torch.bmm(self.dropout(self.attention_weights), values)\n",[49,2690,2691,2713,2718,2747,2771,2784,2788,2793,2798,2803,2808,2845,2862,2867,2893,2905],{"__ignoreMap":59},[63,2692,2693,2696,2700,2702,2706,2708,2711],{"class":65,"line":66},[63,2694,2695],{"class":196},"class",[63,2697,2699],{"class":2698},"skCyd"," DotProductAttention",[63,2701,1078],{"class":2483},[63,2703,2705],{"class":2704},"sDP9b","nn",[63,2707,1142],{"class":2483},[63,2709,2710],{"class":2704},"Module",[63,2712,2496],{"class":2483},[63,2714,2715],{"class":65,"line":88},[63,2716,2717],{"class":2501},"    \"\"\"缩放点积注意力\"\"\"\n",[63,2719,2720,2723,2726,2728,2732,2734,2737,2739,2742,2745],{"class":65,"line":95},[63,2721,2722],{"class":196},"    def",[63,2724,2725],{"class":73}," __init__",[63,2727,1078],{"class":2483},[63,2729,2731],{"class":2730},"sD3jg","self",[63,2733,2490],{"class":2483},[63,2735,2736],{"class":2486},"dropout",[63,2738,2490],{"class":2483},[63,2740,2741],{"class":196},"**",[63,2743,2744],{"class":2486},"kwargs",[63,2746,2496],{"class":2483},[63,2748,2749,2753,2756,2758,2761,2764,2766,2768],{"class":65,"line":108},[63,2750,2752],{"class":2751},"sPGBF","        super",[63,2754,2755],{"class":2483},"(DotProductAttention, ",[63,2757,2731],{"class":81},[63,2759,2760],{"class":2483},").",[63,2762,2763],{"class":73},"__init__",[63,2765,1078],{"class":2483},[63,2767,2741],{"class":196},[63,2769,2770],{"class":2483},"kwargs)\n",[63,2772,2773,2776,2779,2781],{"class":65,"line":113},[63,2774,2775],{"class":81},"        self",[63,2777,2778],{"class":2483},".dropout ",[63,2780,204],{"class":196},[63,2782,2783],{"class":2483}," nn.Dropout(dropout)\n",[63,2785,2786],{"class":65,"line":2548},[63,2787,92],{"emptyLinePlaceholder":91},[63,2789,2790],{"class":65,"line":2556},[63,2791,2792],{"class":2507},"    # queries的形状：(batch_size，查询的个数，d)\n",[63,2794,2795],{"class":65,"line":2567},[63,2796,2797],{"class":2507},"    # keys的形状：(batch_size，“键－值”对的个数，d)\n",[63,2799,2800],{"class":65,"line":2584},[63,2801,2802],{"class":2507},"    # values的形状：(batch_size，“键－值”对的个数，值的维度)\n",[63,2804,2805],{"class":65,"line":2600},[63,2806,2807],{"class":2507},"    # valid_lens的形状:(batch_size，)或者(batch_size，查询的个数)\n",[63,2809,2810,2812,2815,2817,2819,2821,2824,2826,2829,2831,2834,2836,2838,2840,2843],{"class":65,"line":2608},[63,2811,2722],{"class":196},[63,2813,2814],{"class":69}," forward",[63,2816,1078],{"class":2483},[63,2818,2731],{"class":2730},[63,2820,2490],{"class":2483},[63,2822,2823],{"class":2486},"queries",[63,2825,2490],{"class":2483},[63,2827,2828],{"class":2486},"keys",[63,2830,2490],{"class":2483},[63,2832,2833],{"class":2486},"values",[63,2835,2490],{"class":2483},[63,2837,2493],{"class":2486},[63,2839,204],{"class":196},[63,2841,2842],{"class":73},"None",[63,2844,2496],{"class":2483},[63,2846,2847,2850,2852,2855,2857,2859],{"class":65,"line":2625},[63,2848,2849],{"class":2483},"        d ",[63,2851,204],{"class":196},[63,2853,2854],{"class":2483}," queries.shape[",[63,2856,2618],{"class":196},[63,2858,363],{"class":73},[63,2860,2861],{"class":2483},"]\n",[63,2863,2864],{"class":65,"line":2631},[63,2865,2866],{"class":2507},"        # 设置transpose_b=True为了交换keys的最后两个维度\n",[63,2868,2869,2872,2874,2877,2879,2881,2884,2887,2890],{"class":65,"line":2656},[63,2870,2871],{"class":2483},"        scores ",[63,2873,204],{"class":196},[63,2875,2876],{"class":2483}," torch.bmm(queries, keys.transpose(",[63,2878,363],{"class":73},[63,2880,1083],{"class":2483},[63,2882,2883],{"class":73},"2",[63,2885,2886],{"class":2483},")) ",[63,2888,2889],{"class":196},"/",[63,2891,2892],{"class":2483}," math.sqrt(d)\n",[63,2894,2895,2897,2900,2902],{"class":65,"line":2669},[63,2896,2775],{"class":81},[63,2898,2899],{"class":2483},".attention_weights ",[63,2901,204],{"class":196},[63,2903,2904],{"class":2483}," masked_softmax(scores, valid_lens)\n",[63,2906,2908,2910,2913,2915,2918,2920],{"class":65,"line":2907},16,[63,2909,2530],{"class":196},[63,2911,2912],{"class":2483}," torch.bmm(",[63,2914,2731],{"class":81},[63,2916,2917],{"class":2483},".dropout(",[63,2919,2731],{"class":81},[63,2921,2922],{"class":2483},".attention_weights), values)\n",[14,2924,2925,2926,2929,2930,2933],{},"代码中的 ",[49,2927,2928],{},"**kwargs"," 表示接受任意关键字参数，是为了符合 PyTorch ",[49,2931,2932],{},"nn.Module"," 的继承约定。",[30,2935,2936],{"id":2936},"多头注意力",[14,2938,2939],{},"多头注意力（Multi-Head Attention, MHA）允许模型在不同位置上同时关注来自不同表示子空间的信息。如果只用单一注意力头，加权平均的过程会把这些不同的关注模式平均掉，模型的表达力也会随之下降。",[14,2941,2942],{},[323,2943],{"alt":59,"src":2944},"transformer/MHA.png",[14,2946,2947],{},[63,2948,2950,3066],{"className":2949},[342],[63,2951,2953],{"className":2952},[346],[348,2954,2955],{"xmlns":350},[352,2956,2957,3063],{},[355,2958,2959,2983,2985,2987,2989,2991,2993,2995,2997,2999,3015,3017,3032,3034,3037,3039,3053,3055],{},[355,2960,2961,2964,2967,2970,2972,2974,2977,2979,2981],{},[371,2962,2963],{"mathvariant":1053},"M",[371,2965,2966],{"mathvariant":1053},"u",[371,2968,2969],{"mathvariant":1053},"l",[371,2971,1057],{"mathvariant":1053},[371,2973,1069],{"mathvariant":1053},[371,2975,2976],{"mathvariant":1053},"H",[371,2978,1062],{"mathvariant":1053},[371,2980,38],{"mathvariant":1053},[371,2982,373],{"mathvariant":1053},[738,2984,1078],{"stretchy":1077},[371,2986,736],{},[738,2988,1083],{"separator":386},[371,2990,849],{},[738,2992,1083],{"separator":386},[371,2994,949],{},[738,2996,1092],{"stretchy":1077},[738,2998,204],{},[355,3000,3001,3004,3006,3008,3011,3013],{},[371,3002,3003],{"mathvariant":1053},"C",[371,3005,1072],{"mathvariant":1053},[371,3007,612],{"mathvariant":1053},[371,3009,3010],{"mathvariant":1053},"c",[371,3012,38],{"mathvariant":1053},[371,3014,1057],{"mathvariant":1053},[738,3016,1078],{"stretchy":1077},[368,3018,3019,3030],{},[355,3020,3021,3024,3026,3028],{},[371,3022,3023],{"mathvariant":1053},"h",[371,3025,1062],{"mathvariant":1053},[371,3027,38],{"mathvariant":1053},[371,3029,373],{"mathvariant":1053},[361,3031,363],{},[738,3033,1083],{"separator":386},[738,3035,3036],{},"…",[738,3038,1083],{"separator":386},[368,3040,3041,3051],{},[355,3042,3043,3045,3047,3049],{},[371,3044,3023],{"mathvariant":1053},[371,3046,1062],{"mathvariant":1053},[371,3048,38],{"mathvariant":1053},[371,3050,373],{"mathvariant":1053},[371,3052,3023],{},[738,3054,1092],{"stretchy":1077},[742,3056,3057,3060],{},[371,3058,3059],{},"W",[371,3061,3062],{},"O",[378,3064,3065],{"encoding":380},"\\mathrm{MultiHead}(Q, K, V) = \\mathrm{Concat}(\\mathrm{head}_1, \\ldots, \\mathrm{head}_h) W^O",[63,3067,3069,3118],{"className":3068,"ariaHidden":386},[385],[63,3070,3072,3075,3082,3085,3088,3091,3094,3097,3100,3103,3106,3109,3112,3115],{"className":3071},[390],[63,3073],{"className":3074,"style":1155},[394],[63,3076,3078],{"className":3077},[399],[63,3079,3081],{"className":3080},[399,1162],"MultiHead",[63,3083,1078],{"className":3084},[403],[63,3086,736],{"className":3087},[399,472],[63,3089,1083],{"className":3090},[1173],[63,3092],{"className":3093,"style":1177},[776],[63,3095,849],{"className":3096,"style":880},[399,472],[63,3098,1083],{"className":3099},[1173],[63,3101],{"className":3102,"style":1177},[776],[63,3104,949],{"className":3105,"style":979},[399,472],[63,3107,1092],{"className":3108},[593],[63,3110],{"className":3111,"style":777},[776],[63,3113,204],{"className":3114},[781],[63,3116],{"className":3117,"style":777},[776],[63,3119,3121,3125,3132,3135,3181,3184,3187,3190,3193,3196,3199,3242,3245],{"className":3120},[390],[63,3122],{"className":3123,"style":3124},[394],"height:1.0913em;vertical-align:-0.25em;",[63,3126,3128],{"className":3127},[399],[63,3129,3131],{"className":3130},[399,1162],"Concat",[63,3133,1078],{"className":3134},[403],[63,3136,3138,3145],{"className":3137},[399],[63,3139,3141],{"className":3140},[399],[63,3142,3144],{"className":3143},[399,1162],"head",[63,3146,3148],{"className":3147},[476],[63,3149,3151,3173],{"className":3150},[411,412],[63,3152,3154,3170],{"className":3153},[416],[63,3155,3158],{"className":3156,"style":3157},[420],"height:0.3011em;",[63,3159,3161,3164],{"style":3160},"top:-2.55em;margin-right:0.05em;",[63,3162],{"className":3163,"style":817},[428],[63,3165,3167],{"className":3166},[433,434,435,436],[63,3168,363],{"className":3169},[399,436],[63,3171,507],{"className":3172},[506],[63,3174,3176],{"className":3175},[416],[63,3177,3179],{"className":3178,"style":1554},[420],[63,3180],{},[63,3182,1083],{"className":3183},[1173],[63,3185],{"className":3186,"style":1177},[776],[63,3188,3036],{"className":3189},[1222],[63,3191],{"className":3192,"style":1177},[776],[63,3194,1083],{"className":3195},[1173],[63,3197],{"className":3198,"style":1177},[776],[63,3200,3202,3208],{"className":3201},[399],[63,3203,3205],{"className":3204},[399],[63,3206,3144],{"className":3207},[399,1162],[63,3209,3211],{"className":3210},[476],[63,3212,3214,3234],{"className":3213},[411,412],[63,3215,3217,3231],{"className":3216},[416],[63,3218,3220],{"className":3219,"style":1532},[420],[63,3221,3222,3225],{"style":3160},[63,3223],{"className":3224,"style":817},[428],[63,3226,3228],{"className":3227},[433,434,435,436],[63,3229,3023],{"className":3230},[399,472,436],[63,3232,507],{"className":3233},[506],[63,3235,3237],{"className":3236},[416],[63,3238,3240],{"className":3239,"style":1554},[420],[63,3241],{},[63,3243,1092],{"className":3244},[593],[63,3246,3248,3252],{"className":3247},[399],[63,3249,3059],{"className":3250,"style":3251},[399,472],"margin-right:0.13889em;",[63,3253,3255],{"className":3254},[476],[63,3256,3258],{"className":3257},[411],[63,3259,3261],{"className":3260},[416],[63,3262,3265],{"className":3263,"style":3264},[420],"height:0.8413em;",[63,3266,3267,3270],{"style":813},[63,3268],{"className":3269,"style":817},[428],[63,3271,3273],{"className":3272},[433,434,435,436],[63,3274,3062],{"className":3275,"style":3276},[399,472,436],"margin-right:0.02778em;",[14,3278,3279,3280,3641,3642,3891,3892,3891,4141,4392,4393,1142],{},"其中 ",[63,3281,3283,3375],{"className":3282},[342],[63,3284,3286],{"className":3285},[346],[348,3287,3288],{"xmlns":350},[352,3289,3290,3372],{},[355,3291,3292,3306,3308,3328,3330,3332,3340,3342,3346,3348,3356,3358,3360,3362,3370],{},[368,3293,3294,3304],{},[355,3295,3296,3298,3300,3302],{},[371,3297,3023],{"mathvariant":1053},[371,3299,1062],{"mathvariant":1053},[371,3301,38],{"mathvariant":1053},[371,3303,373],{"mathvariant":1053},[371,3305,1069],{},[738,3307,204],{},[355,3309,3310,3312,3314,3316,3318,3320,3322,3324,3326],{},[371,3311,1054],{"mathvariant":1053},[371,3313,1057],{"mathvariant":1053},[371,3315,1057],{"mathvariant":1053},[371,3317,1062],{"mathvariant":1053},[371,3319,612],{"mathvariant":1053},[371,3321,1057],{"mathvariant":1053},[371,3323,1069],{"mathvariant":1053},[371,3325,1072],{"mathvariant":1053},[371,3327,612],{"mathvariant":1053},[738,3329,1078],{"stretchy":1077},[371,3331,736],{},[1870,3333,3334,3336,3338],{},[371,3335,3059],{},[371,3337,1069],{},[371,3339,736],{},[738,3341,1083],{"separator":386},[3343,3344,3345],"mtext",{},"  ",[371,3347,849],{},[1870,3349,3350,3352,3354],{},[371,3351,3059],{},[371,3353,1069],{},[371,3355,849],{},[738,3357,1083],{"separator":386},[3343,3359,3345],{},[371,3361,949],{},[1870,3363,3364,3366,3368],{},[371,3365,3059],{},[371,3367,1069],{},[371,3369,949],{},[738,3371,1092],{"stretchy":1077},[378,3373,3374],{"encoding":380},"\\mathrm{head}_i=\\mathrm{Attention}(QW_i^Q,\\; KW_i^K,\\; VW_i^V)",[63,3376,3378,3436],{"className":3377,"ariaHidden":386},[385],[63,3379,3381,3384,3427,3430,3433],{"className":3380},[390],[63,3382],{"className":3383,"style":1616},[394],[63,3385,3387,3393],{"className":3386},[399],[63,3388,3390],{"className":3389},[399],[63,3391,3144],{"className":3392},[399,1162],[63,3394,3396],{"className":3395},[476],[63,3397,3399,3419],{"className":3398},[411,412],[63,3400,3402,3416],{"className":3401},[416],[63,3403,3405],{"className":3404,"style":2080},[420],[63,3406,3407,3410],{"style":3160},[63,3408],{"className":3409,"style":817},[428],[63,3411,3413],{"className":3412},[433,434,435,436],[63,3414,1069],{"className":3415},[399,472,436],[63,3417,507],{"className":3418},[506],[63,3420,3422],{"className":3421},[416],[63,3423,3425],{"className":3424,"style":1554},[420],[63,3426],{},[63,3428],{"className":3429,"style":777},[776],[63,3431,204],{"className":3432},[781],[63,3434],{"className":3435,"style":777},[776],[63,3437,3439,3443,3449,3452,3455,3510,3513,3516,3519,3522,3575,3578,3581,3584,3587,3638],{"className":3438},[390],[63,3440],{"className":3441,"style":3442},[394],"height:1.2361em;vertical-align:-0.2769em;",[63,3444,3446],{"className":3445},[399],[63,3447,1163],{"className":3448},[399,1162],[63,3450,1078],{"className":3451},[403],[63,3453,736],{"className":3454},[399,472],[63,3456,3458,3461],{"className":3457},[399],[63,3459,3059],{"className":3460,"style":3251},[399,472],[63,3462,3464],{"className":3463},[476],[63,3465,3467,3501],{"className":3466},[411,412],[63,3468,3470,3498],{"className":3469},[416],[63,3471,3474,3486],{"className":3472,"style":3473},[420],"height:0.9592em;",[63,3475,3477,3480],{"style":3476},"top:-2.4231em;margin-left:-0.1389em;margin-right:0.05em;",[63,3478],{"className":3479,"style":817},[428],[63,3481,3483],{"className":3482},[433,434,435,436],[63,3484,1069],{"className":3485},[399,472,436],[63,3487,3489,3492],{"style":3488},"top:-3.1809em;margin-right:0.05em;",[63,3490],{"className":3491,"style":817},[428],[63,3493,3495],{"className":3494},[433,434,435,436],[63,3496,736],{"className":3497},[399,472,436],[63,3499,507],{"className":3500},[506],[63,3502,3504],{"className":3503},[416],[63,3505,3508],{"className":3506,"style":3507},[420],"height:0.2769em;",[63,3509],{},[63,3511,1083],{"className":3512},[1173],[63,3514],{"className":3515,"style":777},[776],[63,3517],{"className":3518,"style":1177},[776],[63,3520,849],{"className":3521,"style":880},[399,472],[63,3523,3525,3528],{"className":3524},[399],[63,3526,3059],{"className":3527,"style":3251},[399,472],[63,3529,3531],{"className":3530},[476],[63,3532,3534,3566],{"className":3533},[411,412],[63,3535,3537,3563],{"className":3536},[416],[63,3538,3540,3552],{"className":3539,"style":3264},[420],[63,3541,3543,3546],{"style":3542},"top:-2.4413em;margin-left:-0.1389em;margin-right:0.05em;",[63,3544],{"className":3545,"style":817},[428],[63,3547,3549],{"className":3548},[433,434,435,436],[63,3550,1069],{"className":3551},[399,472,436],[63,3553,3554,3557],{"style":813},[63,3555],{"className":3556,"style":817},[428],[63,3558,3560],{"className":3559},[433,434,435,436],[63,3561,849],{"className":3562,"style":880},[399,472,436],[63,3564,507],{"className":3565},[506],[63,3567,3569],{"className":3568},[416],[63,3570,3573],{"className":3571,"style":3572},[420],"height:0.2587em;",[63,3574],{},[63,3576,1083],{"className":3577},[1173],[63,3579],{"className":3580,"style":777},[776],[63,3582],{"className":3583,"style":1177},[776],[63,3585,949],{"className":3586,"style":979},[399,472],[63,3588,3590,3593],{"className":3589},[399],[63,3591,3059],{"className":3592,"style":3251},[399,472],[63,3594,3596],{"className":3595},[476],[63,3597,3599,3630],{"className":3598},[411,412],[63,3600,3602,3627],{"className":3601},[416],[63,3603,3605,3616],{"className":3604,"style":3264},[420],[63,3606,3607,3610],{"style":3542},[63,3608],{"className":3609,"style":817},[428],[63,3611,3613],{"className":3612},[433,434,435,436],[63,3614,1069],{"className":3615},[399,472,436],[63,3617,3618,3621],{"style":813},[63,3619],{"className":3620,"style":817},[428],[63,3622,3624],{"className":3623},[433,434,435,436],[63,3625,949],{"className":3626,"style":979},[399,472,436],[63,3628,507],{"className":3629},[506],[63,3631,3633],{"className":3632},[416],[63,3634,3636],{"className":3635,"style":3572},[420],[63,3637],{},[63,3639,1092],{"className":3640},[593],"，投影参数是矩阵\n",[63,3643,3645,3697],{"className":3644},[342],[63,3646,3648],{"className":3647},[346],[348,3649,3650],{"xmlns":350},[352,3651,3652,3694],{},[355,3653,3654,3662,3664],{},[1870,3655,3656,3658,3660],{},[371,3657,3059],{},[371,3659,1069],{},[371,3661,736],{},[738,3663,740],{},[742,3665,3666,3668],{},[371,3667,747],{"mathvariant":746},[355,3669,3670,3686,3688],{},[368,3671,3672,3674],{},[371,3673,373],{},[355,3675,3676,3678,3680,3682,3684],{},[371,3677,643],{"mathvariant":1053},[371,3679,1072],{"mathvariant":1053},[371,3681,373],{"mathvariant":1053},[371,3683,1062],{"mathvariant":1053},[371,3685,2969],{"mathvariant":1053},[738,3687,754],{},[368,3689,3690,3692],{},[371,3691,373],{},[371,3693,376],{},[378,3695,3696],{"encoding":380},"W_i^Q \\in \\mathbb{R}^{d_{\\mathrm{model}} \\times d_k}",[63,3698,3700,3766],{"className":3699,"ariaHidden":386},[385],[63,3701,3703,3706,3757,3760,3763],{"className":3702},[390],[63,3704],{"className":3705,"style":3442},[394],[63,3707,3709,3712],{"className":3708},[399],[63,3710,3059],{"className":3711,"style":3251},[399,472],[63,3713,3715],{"className":3714},[476],[63,3716,3718,3749],{"className":3717},[411,412],[63,3719,3721,3746],{"className":3720},[416],[63,3722,3724,3735],{"className":3723,"style":3473},[420],[63,3725,3726,3729],{"style":3476},[63,3727],{"className":3728,"style":817},[428],[63,3730,3732],{"className":3731},[433,434,435,436],[63,3733,1069],{"className":3734},[399,472,436],[63,3736,3737,3740],{"style":3488},[63,3738],{"className":3739,"style":817},[428],[63,3741,3743],{"className":3742},[433,434,435,436],[63,3744,736],{"className":3745},[399,472,436],[63,3747,507],{"className":3748},[506],[63,3750,3752],{"className":3751},[416],[63,3753,3755],{"className":3754,"style":3507},[420],[63,3756],{},[63,3758],{"className":3759,"style":777},[776],[63,3761,740],{"className":3762},[781],[63,3764],{"className":3765,"style":777},[776],[63,3767,3769,3772],{"className":3768},[390],[63,3770],{"className":3771,"style":791},[394],[63,3773,3775,3778],{"className":3774},[399],[63,3776,747],{"className":3777},[399,798],[63,3779,3781],{"className":3780},[476],[63,3782,3784],{"className":3783},[411],[63,3785,3787],{"className":3786},[416],[63,3788,3790],{"className":3789,"style":791},[420],[63,3791,3792,3795],{"style":813},[63,3793],{"className":3794,"style":817},[428],[63,3796,3798],{"className":3797},[433,434,435,436],[63,3799,3801,3848,3851],{"className":3800},[399,436],[63,3802,3804,3807],{"className":3803},[399,436],[63,3805,373],{"className":3806},[399,472,436],[63,3808,3810],{"className":3809},[476],[63,3811,3813,3840],{"className":3812},[411,412],[63,3814,3816,3837],{"className":3815},[416],[63,3817,3819],{"className":3818,"style":486},[420],[63,3820,3821,3824],{"style":489},[63,3822],{"className":3823,"style":493},[428],[63,3825,3827],{"className":3826},[433,497,498,436],[63,3828,3830],{"className":3829},[399,436],[63,3831,3833],{"className":3832},[399,436],[63,3834,3836],{"className":3835},[399,1162,436],"model",[63,3838,507],{"className":3839},[506],[63,3841,3843],{"className":3842},[416],[63,3844,3846],{"className":3845,"style":514},[420],[63,3847],{},[63,3849,754],{"className":3850},[830,436],[63,3852,3854,3857],{"className":3853},[399,436],[63,3855,373],{"className":3856},[399,472,436],[63,3858,3860],{"className":3859},[476],[63,3861,3863,3883],{"className":3862},[411,412],[63,3864,3866,3880],{"className":3865},[416],[63,3867,3869],{"className":3868,"style":486},[420],[63,3870,3871,3874],{"style":489},[63,3872],{"className":3873,"style":493},[428],[63,3875,3877],{"className":3876},[433,497,498,436],[63,3878,376],{"className":3879,"style":502},[399,472,436],[63,3881,507],{"className":3882},[506],[63,3884,3886],{"className":3885},[416],[63,3887,3889],{"className":3888,"style":514},[420],[63,3890],{},",\n",[63,3893,3895,3947],{"className":3894},[342],[63,3896,3898],{"className":3897},[346],[348,3899,3900],{"xmlns":350},[352,3901,3902,3944],{},[355,3903,3904,3912,3914],{},[1870,3905,3906,3908,3910],{},[371,3907,3059],{},[371,3909,1069],{},[371,3911,849],{},[738,3913,740],{},[742,3915,3916,3918],{},[371,3917,747],{"mathvariant":746},[355,3919,3920,3936,3938],{},[368,3921,3922,3924],{},[371,3923,373],{},[355,3925,3926,3928,3930,3932,3934],{},[371,3927,643],{"mathvariant":1053},[371,3929,1072],{"mathvariant":1053},[371,3931,373],{"mathvariant":1053},[371,3933,1062],{"mathvariant":1053},[371,3935,2969],{"mathvariant":1053},[738,3937,754],{},[368,3939,3940,3942],{},[371,3941,373],{},[371,3943,376],{},[378,3945,3946],{"encoding":380},"W_i^K \\in \\mathbb{R}^{d_{\\mathrm{model}} \\times d_k}",[63,3948,3950,4017],{"className":3949,"ariaHidden":386},[385],[63,3951,3953,3957,4008,4011,4014],{"className":3952},[390],[63,3954],{"className":3955,"style":3956},[394],"height:1.1em;vertical-align:-0.2587em;",[63,3958,3960,3963],{"className":3959},[399],[63,3961,3059],{"className":3962,"style":3251},[399,472],[63,3964,3966],{"className":3965},[476],[63,3967,3969,4000],{"className":3968},[411,412],[63,3970,3972,3997],{"className":3971},[416],[63,3973,3975,3986],{"className":3974,"style":3264},[420],[63,3976,3977,3980],{"style":3542},[63,3978],{"className":3979,"style":817},[428],[63,3981,3983],{"className":3982},[433,434,435,436],[63,3984,1069],{"className":3985},[399,472,436],[63,3987,3988,3991],{"style":813},[63,3989],{"className":3990,"style":817},[428],[63,3992,3994],{"className":3993},[433,434,435,436],[63,3995,849],{"className":3996,"style":880},[399,472,436],[63,3998,507],{"className":3999},[506],[63,4001,4003],{"className":4002},[416],[63,4004,4006],{"className":4005,"style":3572},[420],[63,4007],{},[63,4009],{"className":4010,"style":777},[776],[63,4012,740],{"className":4013},[781],[63,4015],{"className":4016,"style":777},[776],[63,4018,4020,4023],{"className":4019},[390],[63,4021],{"className":4022,"style":791},[394],[63,4024,4026,4029],{"className":4025},[399],[63,4027,747],{"className":4028},[399,798],[63,4030,4032],{"className":4031},[476],[63,4033,4035],{"className":4034},[411],[63,4036,4038],{"className":4037},[416],[63,4039,4041],{"className":4040,"style":791},[420],[63,4042,4043,4046],{"style":813},[63,4044],{"className":4045,"style":817},[428],[63,4047,4049],{"className":4048},[433,434,435,436],[63,4050,4052,4098,4101],{"className":4051},[399,436],[63,4053,4055,4058],{"className":4054},[399,436],[63,4056,373],{"className":4057},[399,472,436],[63,4059,4061],{"className":4060},[476],[63,4062,4064,4090],{"className":4063},[411,412],[63,4065,4067,4087],{"className":4066},[416],[63,4068,4070],{"className":4069,"style":486},[420],[63,4071,4072,4075],{"style":489},[63,4073],{"className":4074,"style":493},[428],[63,4076,4078],{"className":4077},[433,497,498,436],[63,4079,4081],{"className":4080},[399,436],[63,4082,4084],{"className":4083},[399,436],[63,4085,3836],{"className":4086},[399,1162,436],[63,4088,507],{"className":4089},[506],[63,4091,4093],{"className":4092},[416],[63,4094,4096],{"className":4095,"style":514},[420],[63,4097],{},[63,4099,754],{"className":4100},[830,436],[63,4102,4104,4107],{"className":4103},[399,436],[63,4105,373],{"className":4106},[399,472,436],[63,4108,4110],{"className":4109},[476],[63,4111,4113,4133],{"className":4112},[411,412],[63,4114,4116,4130],{"className":4115},[416],[63,4117,4119],{"className":4118,"style":486},[420],[63,4120,4121,4124],{"style":489},[63,4122],{"className":4123,"style":493},[428],[63,4125,4127],{"className":4126},[433,497,498,436],[63,4128,376],{"className":4129,"style":502},[399,472,436],[63,4131,507],{"className":4132},[506],[63,4134,4136],{"className":4135},[416],[63,4137,4139],{"className":4138,"style":514},[420],[63,4140],{},[63,4142,4144,4196],{"className":4143},[342],[63,4145,4147],{"className":4146},[346],[348,4148,4149],{"xmlns":350},[352,4150,4151,4193],{},[355,4152,4153,4161,4163],{},[1870,4154,4155,4157,4159],{},[371,4156,3059],{},[371,4158,1069],{},[371,4160,949],{},[738,4162,740],{},[742,4164,4165,4167],{},[371,4166,747],{"mathvariant":746},[355,4168,4169,4185,4187],{},[368,4170,4171,4173],{},[371,4172,373],{},[355,4174,4175,4177,4179,4181,4183],{},[371,4176,643],{"mathvariant":1053},[371,4178,1072],{"mathvariant":1053},[371,4180,373],{"mathvariant":1053},[371,4182,1062],{"mathvariant":1053},[371,4184,2969],{"mathvariant":1053},[738,4186,754],{},[368,4188,4189,4191],{},[371,4190,373],{},[371,4192,703],{},[378,4194,4195],{"encoding":380},"W_i^V \\in \\mathbb{R}^{d_{\\mathrm{model}} \\times d_v}",[63,4197,4199,4265],{"className":4198,"ariaHidden":386},[385],[63,4200,4202,4205,4256,4259,4262],{"className":4201},[390],[63,4203],{"className":4204,"style":3956},[394],[63,4206,4208,4211],{"className":4207},[399],[63,4209,3059],{"className":4210,"style":3251},[399,472],[63,4212,4214],{"className":4213},[476],[63,4215,4217,4248],{"className":4216},[411,412],[63,4218,4220,4245],{"className":4219},[416],[63,4221,4223,4234],{"className":4222,"style":3264},[420],[63,4224,4225,4228],{"style":3542},[63,4226],{"className":4227,"style":817},[428],[63,4229,4231],{"className":4230},[433,434,435,436],[63,4232,1069],{"className":4233},[399,472,436],[63,4235,4236,4239],{"style":813},[63,4237],{"className":4238,"style":817},[428],[63,4240,4242],{"className":4241},[433,434,435,436],[63,4243,949],{"className":4244,"style":979},[399,472,436],[63,4246,507],{"className":4247},[506],[63,4249,4251],{"className":4250},[416],[63,4252,4254],{"className":4253,"style":3572},[420],[63,4255],{},[63,4257],{"className":4258,"style":777},[776],[63,4260,740],{"className":4261},[781],[63,4263],{"className":4264,"style":777},[776],[63,4266,4268,4271],{"className":4267},[390],[63,4269],{"className":4270,"style":791},[394],[63,4272,4274,4277],{"className":4273},[399],[63,4275,747],{"className":4276},[399,798],[63,4278,4280],{"className":4279},[476],[63,4281,4283],{"className":4282},[411],[63,4284,4286],{"className":4285},[416],[63,4287,4289],{"className":4288,"style":791},[420],[63,4290,4291,4294],{"style":813},[63,4292],{"className":4293,"style":817},[428],[63,4295,4297],{"className":4296},[433,434,435,436],[63,4298,4300,4346,4349],{"className":4299},[399,436],[63,4301,4303,4306],{"className":4302},[399,436],[63,4304,373],{"className":4305},[399,472,436],[63,4307,4309],{"className":4308},[476],[63,4310,4312,4338],{"className":4311},[411,412],[63,4313,4315,4335],{"className":4314},[416],[63,4316,4318],{"className":4317,"style":486},[420],[63,4319,4320,4323],{"style":489},[63,4321],{"className":4322,"style":493},[428],[63,4324,4326],{"className":4325},[433,497,498,436],[63,4327,4329],{"className":4328},[399,436],[63,4330,4332],{"className":4331},[399,436],[63,4333,3836],{"className":4334},[399,1162,436],[63,4336,507],{"className":4337},[506],[63,4339,4341],{"className":4340},[416],[63,4342,4344],{"className":4343,"style":514},[420],[63,4345],{},[63,4347,754],{"className":4348},[830,436],[63,4350,4352,4355],{"className":4351},[399,436],[63,4353,373],{"className":4354},[399,472,436],[63,4356,4358],{"className":4357},[476],[63,4359,4361,4383],{"className":4360},[411,412],[63,4362,4364,4380],{"className":4363},[416],[63,4365,4368],{"className":4366,"style":4367},[420],"height:0.1645em;",[63,4369,4371,4374],{"style":4370},"top:-2.357em;margin-left:0em;margin-right:0.0714em;",[63,4372],{"className":4373,"style":493},[428],[63,4375,4377],{"className":4376},[433,497,498,436],[63,4378,703],{"className":4379,"style":718},[399,472,436],[63,4381,507],{"className":4382},[506],[63,4384,4386],{"className":4385},[416],[63,4387,4390],{"className":4388,"style":4389},[420],"height:0.143em;",[63,4391],{},"\n和\n",[63,4394,4396,4448],{"className":4395},[342],[63,4397,4399],{"className":4398},[346],[348,4400,4401],{"xmlns":350},[352,4402,4403,4445],{},[355,4404,4405,4411,4413],{},[742,4406,4407,4409],{},[371,4408,3059],{},[371,4410,3062],{},[738,4412,740],{},[742,4414,4415,4417],{},[371,4416,747],{"mathvariant":746},[355,4418,4419,4421,4427,4429],{},[371,4420,3023],{},[368,4422,4423,4425],{},[371,4424,373],{},[371,4426,703],{},[738,4428,754],{},[368,4430,4431,4433],{},[371,4432,373],{},[355,4434,4435,4437,4439,4441,4443],{},[371,4436,643],{"mathvariant":1053},[371,4438,1072],{"mathvariant":1053},[371,4440,373],{"mathvariant":1053},[371,4442,1062],{"mathvariant":1053},[371,4444,2969],{"mathvariant":1053},[378,4446,4447],{"encoding":380},"W^O \\in \\mathbb{R}^{h d_v \\times d_{\\mathrm{model}}}",[63,4449,4451,4496],{"className":4450,"ariaHidden":386},[385],[63,4452,4454,4458,4487,4490,4493],{"className":4453},[390],[63,4455],{"className":4456,"style":4457},[394],"height:0.8804em;vertical-align:-0.0391em;",[63,4459,4461,4464],{"className":4460},[399],[63,4462,3059],{"className":4463,"style":3251},[399,472],[63,4465,4467],{"className":4466},[476],[63,4468,4470],{"className":4469},[411],[63,4471,4473],{"className":4472},[416],[63,4474,4476],{"className":4475,"style":3264},[420],[63,4477,4478,4481],{"style":813},[63,4479],{"className":4480,"style":817},[428],[63,4482,4484],{"className":4483},[433,434,435,436],[63,4485,3062],{"className":4486,"style":3276},[399,472,436],[63,4488],{"className":4489,"style":777},[776],[63,4491,740],{"className":4492},[781],[63,4494],{"className":4495,"style":777},[776],[63,4497,4499,4502],{"className":4498},[390],[63,4500],{"className":4501,"style":791},[394],[63,4503,4505,4508],{"className":4504},[399],[63,4506,747],{"className":4507},[399,798],[63,4509,4511],{"className":4510},[476],[63,4512,4514],{"className":4513},[411],[63,4515,4517],{"className":4516},[416],[63,4518,4520],{"className":4519,"style":791},[420],[63,4521,4522,4525],{"style":813},[63,4523],{"className":4524,"style":817},[428],[63,4526,4528],{"className":4527},[433,434,435,436],[63,4529,4531,4534,4574,4577],{"className":4530},[399,436],[63,4532,3023],{"className":4533},[399,472,436],[63,4535,4537,4540],{"className":4536},[399,436],[63,4538,373],{"className":4539},[399,472,436],[63,4541,4543],{"className":4542},[476],[63,4544,4546,4566],{"className":4545},[411,412],[63,4547,4549,4563],{"className":4548},[416],[63,4550,4552],{"className":4551,"style":4367},[420],[63,4553,4554,4557],{"style":4370},[63,4555],{"className":4556,"style":493},[428],[63,4558,4560],{"className":4559},[433,497,498,436],[63,4561,703],{"className":4562,"style":718},[399,472,436],[63,4564,507],{"className":4565},[506],[63,4567,4569],{"className":4568},[416],[63,4570,4572],{"className":4571,"style":4389},[420],[63,4573],{},[63,4575,754],{"className":4576},[830,436],[63,4578,4580,4583],{"className":4579},[399,436],[63,4581,373],{"className":4582},[399,472,436],[63,4584,4586],{"className":4585},[476],[63,4587,4589,4615],{"className":4588},[411,412],[63,4590,4592,4612],{"className":4591},[416],[63,4593,4595],{"className":4594,"style":486},[420],[63,4596,4597,4600],{"style":489},[63,4598],{"className":4599,"style":493},[428],[63,4601,4603],{"className":4602},[433,497,498,436],[63,4604,4606],{"className":4605},[399,436],[63,4607,4609],{"className":4608},[399,436],[63,4610,3836],{"className":4611},[399,1162,436],[63,4613,507],{"className":4614},[506],[63,4616,4618],{"className":4617},[416],[63,4619,4621],{"className":4620,"style":514},[420],[63,4622],{},[14,4624,4625,4626,4629,4630,4633,4634,4637,4638,4641],{},"为了让多个头能够并行计算，我们定义两个变换函数 ",[49,4627,4628],{},"transpose_qkv"," 与 ",[49,4631,4632],{},"transpose_output","，前者将 ",[49,4635,4636],{},"(batch, seq_len, num_hiddens)"," 重排为 ",[49,4639,4640],{},"(batch * num_heads, seq_len, num_hiddens / num_heads)","，后者负责把结果还原回原始形状。",[54,4643,4645],{"className":2468,"code":4644,"language":2470,"meta":59,"style":59},"def transpose_qkv(X, num_heads):\n    \"\"\"为了多注意力头的并行计算而变换形状\"\"\"\n    # 输入X的形状:(batch_size，查询或者“键－值”对的个数，num_hiddens)\n    # 输出X的形状:(batch_size，查询或者“键－值”对的个数，num_heads，num_hiddens/num_heads)\n    X = X.reshape(X.shape[0], X.shape[1], num_heads, -1)\n\n    # 输出X的形状:(batch_size，num_heads，查询或者“键－值”对的个数, num_hiddens/num_heads)\n    X = X.permute(0, 2, 1, 3)\n\n    # 最终输出的形状:(batch_size*num_heads,查询或者“键－值”对的个数, num_hiddens/num_heads)\n    return X.reshape(-1, X.shape[2], X.shape[3])\n\ndef transpose_output(X, num_heads):\n    \"\"\"逆转transpose_qkv函数的操作\"\"\"\n    X = X.reshape(-1, num_heads, X.shape[1], X.shape[2])\n    X = X.permute(0, 2, 1, 3)\n    return X.reshape(X.shape[0], X.shape[1], -1)\n",[49,4646,4647,4665,4670,4675,4680,4706,4710,4715,4741,4745,4750,4773,4777,4794,4799,4822,4846],{"__ignoreMap":59},[63,4648,4649,4651,4654,4656,4658,4660,4663],{"class":65,"line":66},[63,4650,2477],{"class":196},[63,4652,4653],{"class":69}," transpose_qkv",[63,4655,1078],{"class":2483},[63,4657,2487],{"class":2486},[63,4659,2490],{"class":2483},[63,4661,4662],{"class":2486},"num_heads",[63,4664,2496],{"class":2483},[63,4666,4667],{"class":65,"line":88},[63,4668,4669],{"class":2501},"    \"\"\"为了多注意力头的并行计算而变换形状\"\"\"\n",[63,4671,4672],{"class":65,"line":95},[63,4673,4674],{"class":2507},"    # 输入X的形状:(batch_size，查询或者“键－值”对的个数，num_hiddens)\n",[63,4676,4677],{"class":65,"line":108},[63,4678,4679],{"class":2507},"    # 输出X的形状:(batch_size，查询或者“键－值”对的个数，num_heads，num_hiddens/num_heads)\n",[63,4681,4682,4685,4687,4690,4692,4695,4697,4700,4702,4704],{"class":65,"line":113},[63,4683,4684],{"class":2483},"    X ",[63,4686,204],{"class":196},[63,4688,4689],{"class":2483}," X.reshape(X.shape[",[63,4691,1802],{"class":73},[63,4693,4694],{"class":2483},"], X.shape[",[63,4696,363],{"class":73},[63,4698,4699],{"class":2483},"], num_heads, ",[63,4701,2618],{"class":196},[63,4703,363],{"class":73},[63,4705,2545],{"class":2483},[63,4707,4708],{"class":65,"line":2548},[63,4709,92],{"emptyLinePlaceholder":91},[63,4711,4712],{"class":65,"line":2556},[63,4713,4714],{"class":2507},"    # 输出X的形状:(batch_size，num_heads，查询或者“键－值”对的个数, num_hiddens/num_heads)\n",[63,4716,4717,4719,4721,4724,4726,4728,4730,4732,4734,4736,4739],{"class":65,"line":2567},[63,4718,4684],{"class":2483},[63,4720,204],{"class":196},[63,4722,4723],{"class":2483}," X.permute(",[63,4725,1802],{"class":73},[63,4727,2490],{"class":2483},[63,4729,2883],{"class":73},[63,4731,2490],{"class":2483},[63,4733,363],{"class":73},[63,4735,2490],{"class":2483},[63,4737,4738],{"class":73},"3",[63,4740,2545],{"class":2483},[63,4742,4743],{"class":65,"line":2584},[63,4744,92],{"emptyLinePlaceholder":91},[63,4746,4747],{"class":65,"line":2600},[63,4748,4749],{"class":2507},"    # 最终输出的形状:(batch_size*num_heads,查询或者“键－值”对的个数, num_hiddens/num_heads)\n",[63,4751,4752,4755,4758,4760,4762,4765,4767,4769,4771],{"class":65,"line":2608},[63,4753,4754],{"class":196},"    return",[63,4756,4757],{"class":2483}," X.reshape(",[63,4759,2618],{"class":196},[63,4761,363],{"class":73},[63,4763,4764],{"class":2483},", X.shape[",[63,4766,2883],{"class":73},[63,4768,4694],{"class":2483},[63,4770,4738],{"class":73},[63,4772,2597],{"class":2483},[63,4774,4775],{"class":65,"line":2625},[63,4776,92],{"emptyLinePlaceholder":91},[63,4778,4779,4781,4784,4786,4788,4790,4792],{"class":65,"line":2631},[63,4780,2477],{"class":196},[63,4782,4783],{"class":69}," transpose_output",[63,4785,1078],{"class":2483},[63,4787,2487],{"class":2486},[63,4789,2490],{"class":2483},[63,4791,4662],{"class":2486},[63,4793,2496],{"class":2483},[63,4795,4796],{"class":65,"line":2656},[63,4797,4798],{"class":2501},"    \"\"\"逆转transpose_qkv函数的操作\"\"\"\n",[63,4800,4801,4803,4805,4807,4809,4811,4814,4816,4818,4820],{"class":65,"line":2669},[63,4802,4684],{"class":2483},[63,4804,204],{"class":196},[63,4806,4757],{"class":2483},[63,4808,2618],{"class":196},[63,4810,363],{"class":73},[63,4812,4813],{"class":2483},", num_heads, X.shape[",[63,4815,363],{"class":73},[63,4817,4694],{"class":2483},[63,4819,2883],{"class":73},[63,4821,2597],{"class":2483},[63,4823,4824,4826,4828,4830,4832,4834,4836,4838,4840,4842,4844],{"class":65,"line":2907},[63,4825,4684],{"class":2483},[63,4827,204],{"class":196},[63,4829,4723],{"class":2483},[63,4831,1802],{"class":73},[63,4833,2490],{"class":2483},[63,4835,2883],{"class":73},[63,4837,2490],{"class":2483},[63,4839,363],{"class":73},[63,4841,2490],{"class":2483},[63,4843,4738],{"class":73},[63,4845,2545],{"class":2483},[63,4847,4849,4851,4853,4855,4857,4859,4862,4864,4866],{"class":65,"line":4848},17,[63,4850,4754],{"class":196},[63,4852,4689],{"class":2483},[63,4854,1802],{"class":73},[63,4856,4694],{"class":2483},[63,4858,363],{"class":73},[63,4860,4861],{"class":2483},"], ",[63,4863,2618],{"class":196},[63,4865,363],{"class":73},[63,4867,2545],{"class":2483},[54,4869,4871],{"className":2468,"code":4870,"language":2470,"meta":59,"style":59},"#@save\nclass MultiHeadAttention(nn.Module):\n    \"\"\"多头注意力\"\"\"\n    def __init__(self, key_size, query_size, value_size, num_hiddens,\n                 num_heads, dropout, bias=False, **kwargs):\n        super(MultiHeadAttention, self).__init__(**kwargs)\n        self.num_heads = num_heads\n        self.attention = d2l.DotProductAttention(dropout)\n        self.W_q = nn.Linear(query_size, num_hiddens, bias=bias)\n        self.W_k = nn.Linear(key_size, num_hiddens, bias=bias)\n        self.W_v = nn.Linear(value_size, num_hiddens, bias=bias)\n        self.W_o = nn.Linear(num_hiddens, num_hiddens, bias=bias)\n\n    def forward(self, queries, keys, values, valid_lens):\n        # queries，keys，values的形状:\n        # (batch_size，查询或者“键－值”对的个数，num_hiddens)\n        # valid_lens　的形状:\n        # (batch_size，)或(batch_size，查询的个数)\n        # 经过变换后，输出的queries，keys，values 的形状:\n        # (batch_size*num_heads，查询或者“键－值”对的个数，num_hiddens/num_heads)\n        queries = transpose_qkv(self.W_q(queries), self.num_heads)\n        keys = transpose_qkv(self.W_k(keys), self.num_heads)\n        values = transpose_qkv(self.W_v(values), self.num_heads)\n\n        if valid_lens is not None:\n            # 在轴0，将第一项（标量或者矢量）复制num_heads次，\n            # 然后如此复制第二项，然后诸如此类。\n            valid_lens = torch.repeat_interleave(\n                valid_lens, repeats=self.num_heads, dim=0)\n\n        # output的形状:(batch_size*num_heads，查询的个数， num_hiddens/num_heads)\n        output = self.attention(queries, keys, values, valid_lens)\n\n        # output_concat的形状:(batch_size，查询的个数，num_hiddens)\n        output_concat = transpose_output(output, self.num_heads)\n        return self.W_o(output_concat)\n",[49,4872,4873,4878,4895,4900,4932,4959,4978,4990,5002,5021,5039,5057,5075,5079,5107,5112,5117,5122,5128,5134,5140,5161,5180,5199,5204,5220,5226,5232,5242,5266,5271,5277,5291,5296,5302,5317],{"__ignoreMap":59},[63,4874,4875],{"class":65,"line":66},[63,4876,4877],{"class":2507},"#@save\n",[63,4879,4880,4882,4885,4887,4889,4891,4893],{"class":65,"line":88},[63,4881,2695],{"class":196},[63,4883,4884],{"class":2698}," MultiHeadAttention",[63,4886,1078],{"class":2483},[63,4888,2705],{"class":2704},[63,4890,1142],{"class":2483},[63,4892,2710],{"class":2704},[63,4894,2496],{"class":2483},[63,4896,4897],{"class":65,"line":95},[63,4898,4899],{"class":2501},"    \"\"\"多头注意力\"\"\"\n",[63,4901,4902,4904,4906,4908,4910,4912,4915,4917,4920,4922,4925,4927,4930],{"class":65,"line":108},[63,4903,2722],{"class":196},[63,4905,2725],{"class":73},[63,4907,1078],{"class":2483},[63,4909,2731],{"class":2730},[63,4911,2490],{"class":2483},[63,4913,4914],{"class":2486},"key_size",[63,4916,2490],{"class":2483},[63,4918,4919],{"class":2486},"query_size",[63,4921,2490],{"class":2483},[63,4923,4924],{"class":2486},"value_size",[63,4926,2490],{"class":2483},[63,4928,4929],{"class":2486},"num_hiddens",[63,4931,3891],{"class":2483},[63,4933,4934,4937,4939,4941,4943,4946,4948,4951,4953,4955,4957],{"class":65,"line":113},[63,4935,4936],{"class":2486},"                 num_heads",[63,4938,2490],{"class":2483},[63,4940,2736],{"class":2486},[63,4942,2490],{"class":2483},[63,4944,4945],{"class":2486},"bias",[63,4947,204],{"class":196},[63,4949,4950],{"class":73},"False",[63,4952,2490],{"class":2483},[63,4954,2741],{"class":196},[63,4956,2744],{"class":2486},[63,4958,2496],{"class":2483},[63,4960,4961,4963,4966,4968,4970,4972,4974,4976],{"class":65,"line":2548},[63,4962,2752],{"class":2751},[63,4964,4965],{"class":2483},"(MultiHeadAttention, ",[63,4967,2731],{"class":81},[63,4969,2760],{"class":2483},[63,4971,2763],{"class":73},[63,4973,1078],{"class":2483},[63,4975,2741],{"class":196},[63,4977,2770],{"class":2483},[63,4979,4980,4982,4985,4987],{"class":65,"line":2556},[63,4981,2775],{"class":81},[63,4983,4984],{"class":2483},".num_heads ",[63,4986,204],{"class":196},[63,4988,4989],{"class":2483}," num_heads\n",[63,4991,4992,4994,4997,4999],{"class":65,"line":2567},[63,4993,2775],{"class":81},[63,4995,4996],{"class":2483},".attention ",[63,4998,204],{"class":196},[63,5000,5001],{"class":2483}," d2l.DotProductAttention(dropout)\n",[63,5003,5004,5006,5009,5011,5014,5016,5018],{"class":65,"line":2584},[63,5005,2775],{"class":81},[63,5007,5008],{"class":2483},".W_q ",[63,5010,204],{"class":196},[63,5012,5013],{"class":2483}," nn.Linear(query_size, num_hiddens, ",[63,5015,4945],{"class":2536},[63,5017,204],{"class":196},[63,5019,5020],{"class":2483},"bias)\n",[63,5022,5023,5025,5028,5030,5033,5035,5037],{"class":65,"line":2600},[63,5024,2775],{"class":81},[63,5026,5027],{"class":2483},".W_k ",[63,5029,204],{"class":196},[63,5031,5032],{"class":2483}," nn.Linear(key_size, num_hiddens, ",[63,5034,4945],{"class":2536},[63,5036,204],{"class":196},[63,5038,5020],{"class":2483},[63,5040,5041,5043,5046,5048,5051,5053,5055],{"class":65,"line":2608},[63,5042,2775],{"class":81},[63,5044,5045],{"class":2483},".W_v ",[63,5047,204],{"class":196},[63,5049,5050],{"class":2483}," nn.Linear(value_size, num_hiddens, ",[63,5052,4945],{"class":2536},[63,5054,204],{"class":196},[63,5056,5020],{"class":2483},[63,5058,5059,5061,5064,5066,5069,5071,5073],{"class":65,"line":2625},[63,5060,2775],{"class":81},[63,5062,5063],{"class":2483},".W_o ",[63,5065,204],{"class":196},[63,5067,5068],{"class":2483}," nn.Linear(num_hiddens, num_hiddens, ",[63,5070,4945],{"class":2536},[63,5072,204],{"class":196},[63,5074,5020],{"class":2483},[63,5076,5077],{"class":65,"line":2631},[63,5078,92],{"emptyLinePlaceholder":91},[63,5080,5081,5083,5085,5087,5089,5091,5093,5095,5097,5099,5101,5103,5105],{"class":65,"line":2656},[63,5082,2722],{"class":196},[63,5084,2814],{"class":69},[63,5086,1078],{"class":2483},[63,5088,2731],{"class":2730},[63,5090,2490],{"class":2483},[63,5092,2823],{"class":2486},[63,5094,2490],{"class":2483},[63,5096,2828],{"class":2486},[63,5098,2490],{"class":2483},[63,5100,2833],{"class":2486},[63,5102,2490],{"class":2483},[63,5104,2493],{"class":2486},[63,5106,2496],{"class":2483},[63,5108,5109],{"class":65,"line":2669},[63,5110,5111],{"class":2507},"        # queries，keys，values的形状:\n",[63,5113,5114],{"class":65,"line":2907},[63,5115,5116],{"class":2507},"        # (batch_size，查询或者“键－值”对的个数，num_hiddens)\n",[63,5118,5119],{"class":65,"line":4848},[63,5120,5121],{"class":2507},"        # valid_lens　的形状:\n",[63,5123,5125],{"class":65,"line":5124},18,[63,5126,5127],{"class":2507},"        # (batch_size，)或(batch_size，查询的个数)\n",[63,5129,5131],{"class":65,"line":5130},19,[63,5132,5133],{"class":2507},"        # 经过变换后，输出的queries，keys，values 的形状:\n",[63,5135,5137],{"class":65,"line":5136},20,[63,5138,5139],{"class":2507},"        # (batch_size*num_heads，查询或者“键－值”对的个数，num_hiddens/num_heads)\n",[63,5141,5143,5146,5148,5151,5153,5156,5158],{"class":65,"line":5142},21,[63,5144,5145],{"class":2483},"        queries ",[63,5147,204],{"class":196},[63,5149,5150],{"class":2483}," transpose_qkv(",[63,5152,2731],{"class":81},[63,5154,5155],{"class":2483},".W_q(queries), ",[63,5157,2731],{"class":81},[63,5159,5160],{"class":2483},".num_heads)\n",[63,5162,5164,5167,5169,5171,5173,5176,5178],{"class":65,"line":5163},22,[63,5165,5166],{"class":2483},"        keys ",[63,5168,204],{"class":196},[63,5170,5150],{"class":2483},[63,5172,2731],{"class":81},[63,5174,5175],{"class":2483},".W_k(keys), ",[63,5177,2731],{"class":81},[63,5179,5160],{"class":2483},[63,5181,5183,5186,5188,5190,5192,5195,5197],{"class":65,"line":5182},23,[63,5184,5185],{"class":2483},"        values ",[63,5187,204],{"class":196},[63,5189,5150],{"class":2483},[63,5191,2731],{"class":81},[63,5193,5194],{"class":2483},".W_v(values), ",[63,5196,2731],{"class":81},[63,5198,5160],{"class":2483},[63,5200,5202],{"class":65,"line":5201},24,[63,5203,92],{"emptyLinePlaceholder":91},[63,5205,5207,5209,5211,5213,5216,5218],{"class":65,"line":5206},25,[63,5208,2570],{"class":196},[63,5210,2516],{"class":2483},[63,5212,2519],{"class":196},[63,5214,5215],{"class":196}," not",[63,5217,2522],{"class":73},[63,5219,2525],{"class":2483},[63,5221,5223],{"class":65,"line":5222},26,[63,5224,5225],{"class":2507},"            # 在轴0，将第一项（标量或者矢量）复制num_heads次，\n",[63,5227,5229],{"class":65,"line":5228},27,[63,5230,5231],{"class":2507},"            # 然后如此复制第二项，然后诸如此类。\n",[63,5233,5235,5237,5239],{"class":65,"line":5234},28,[63,5236,2587],{"class":2483},[63,5238,204],{"class":196},[63,5240,5241],{"class":2483}," torch.repeat_interleave(\n",[63,5243,5245,5248,5251,5253,5255,5258,5260,5262,5264],{"class":65,"line":5244},29,[63,5246,5247],{"class":2483},"                valid_lens, ",[63,5249,5250],{"class":2536},"repeats",[63,5252,204],{"class":196},[63,5254,2731],{"class":81},[63,5256,5257],{"class":2483},".num_heads, ",[63,5259,2537],{"class":2536},[63,5261,204],{"class":196},[63,5263,1802],{"class":73},[63,5265,2545],{"class":2483},[63,5267,5269],{"class":65,"line":5268},30,[63,5270,92],{"emptyLinePlaceholder":91},[63,5272,5274],{"class":65,"line":5273},31,[63,5275,5276],{"class":2507},"        # output的形状:(batch_size*num_heads，查询的个数， num_hiddens/num_heads)\n",[63,5278,5280,5283,5285,5288],{"class":65,"line":5279},32,[63,5281,5282],{"class":2483},"        output ",[63,5284,204],{"class":196},[63,5286,5287],{"class":81}," self",[63,5289,5290],{"class":2483},".attention(queries, keys, values, valid_lens)\n",[63,5292,5294],{"class":65,"line":5293},33,[63,5295,92],{"emptyLinePlaceholder":91},[63,5297,5299],{"class":65,"line":5298},34,[63,5300,5301],{"class":2507},"        # output_concat的形状:(batch_size，查询的个数，num_hiddens)\n",[63,5303,5305,5308,5310,5313,5315],{"class":65,"line":5304},35,[63,5306,5307],{"class":2483},"        output_concat ",[63,5309,204],{"class":196},[63,5311,5312],{"class":2483}," transpose_output(output, ",[63,5314,2731],{"class":81},[63,5316,5160],{"class":2483},[63,5318,5320,5322,5324],{"class":65,"line":5319},36,[63,5321,2530],{"class":196},[63,5323,5287],{"class":81},[63,5325,5326],{"class":2483},".W_o(output_concat)\n",[10,5328,5329],{"id":5329},"位置编码",[14,5331,5332,5333,5410],{},"RNN 是按时间步逐个处理 token 的，天然带有顺序信息；而注意力机制为了支持并行计算放弃了顺序处理，因而失去了位置信息。为了让模型仍能感知到 token 在序列中的位置，我们需要把\"位置\"显式地注入到输入表示中——这就是位置编码（Positional Encoding）。位置编码与词嵌入的维度相同，均为 ",[63,5334,5336,5354],{"className":5335},[342],[63,5337,5339],{"className":5338},[346],[348,5340,5341],{"xmlns":350},[352,5342,5343,5351],{},[355,5344,5345],{},[368,5346,5347,5349],{},[371,5348,373],{},[3343,5350,3836],{},[378,5352,5353],{"encoding":380},"d_{\\text{model}}",[63,5355,5357],{"className":5356,"ariaHidden":386},[385],[63,5358,5360,5363],{"className":5359},[390],[63,5361],{"className":5362,"style":1616},[394],[63,5364,5366,5369],{"className":5365},[399],[63,5367,373],{"className":5368},[399,472],[63,5370,5372],{"className":5371},[476],[63,5373,5375,5402],{"className":5374},[411,412],[63,5376,5378,5399],{"className":5377},[416],[63,5379,5381],{"className":5380,"style":1532},[420],[63,5382,5383,5386],{"style":1535},[63,5384],{"className":5385,"style":817},[428],[63,5387,5389],{"className":5388},[433,434,435,436],[63,5390,5392],{"className":5391},[399,436],[63,5393,5396],{"className":5394},[399,5395,436],"text",[63,5397,3836],{"className":5398},[399,436],[63,5400,507],{"className":5401},[506],[63,5403,5405],{"className":5404},[416],[63,5406,5408],{"className":5407,"style":1554},[420],[63,5409],{},"，因此可以直接相加。它既可以通过参数学习得到，也可以用固定的公式直接计算。",[14,5412,5413],{},"这里我们使用基于正弦和余弦函数的固定位置编码。",[14,5415,5416,5417,5563,5564,5592,5593,5668,5669,5822,5823,5884,5885,5914,5915,4629,5952,6008],{},"设输入表示 ",[63,5418,5420,5453],{"className":5419},[342],[63,5421,5423],{"className":5422},[346],[348,5424,5425],{"xmlns":350},[352,5426,5427,5450],{},[355,5428,5429,5432,5434],{},[371,5430,2487],{"mathvariant":5431},"bold",[738,5433,740],{},[742,5435,5436,5438],{},[371,5437,747],{"mathvariant":746},[355,5439,5440,5442,5444],{},[371,5441,612],{},[738,5443,754],{},[368,5445,5446,5448],{},[371,5447,373],{},[3343,5449,3836],{},[378,5451,5452],{"encoding":380},"\\mathbf{X} \\in \\mathbb{R}^{n \\times d_{\\text{model}}}",[63,5454,5456,5476],{"className":5455,"ariaHidden":386},[385],[63,5457,5459,5463,5467,5470,5473],{"className":5458},[390],[63,5460],{"className":5461,"style":5462},[394],"height:0.7252em;vertical-align:-0.0391em;",[63,5464,2487],{"className":5465},[399,5466],"mathbf",[63,5468],{"className":5469,"style":777},[776],[63,5471,740],{"className":5472},[781],[63,5474],{"className":5475,"style":777},[776],[63,5477,5479,5482],{"className":5478},[390],[63,5480],{"className":5481,"style":791},[394],[63,5483,5485,5488],{"className":5484},[399],[63,5486,747],{"className":5487},[399,798],[63,5489,5491],{"className":5490},[476],[63,5492,5494],{"className":5493},[411],[63,5495,5497],{"className":5496},[416],[63,5498,5500],{"className":5499,"style":791},[420],[63,5501,5502,5505],{"style":813},[63,5503],{"className":5504,"style":817},[428],[63,5506,5508],{"className":5507},[433,434,435,436],[63,5509,5511,5514,5517],{"className":5510},[399,436],[63,5512,612],{"className":5513},[399,472,436],[63,5515,754],{"className":5516},[830,436],[63,5518,5520,5523],{"className":5519},[399,436],[63,5521,373],{"className":5522},[399,472,436],[63,5524,5526],{"className":5525},[476],[63,5527,5529,5555],{"className":5528},[411,412],[63,5530,5532,5552],{"className":5531},[416],[63,5533,5535],{"className":5534,"style":486},[420],[63,5536,5537,5540],{"style":489},[63,5538],{"className":5539,"style":493},[428],[63,5541,5543],{"className":5542},[433,497,498,436],[63,5544,5546],{"className":5545},[399,436],[63,5547,5549],{"className":5548},[399,5395,436],[63,5550,3836],{"className":5551},[399,436],[63,5553,507],{"className":5554},[506],[63,5556,5558],{"className":5557},[416],[63,5559,5561],{"className":5560,"style":514},[420],[63,5562],{}," 包含一段长度为 ",[63,5565,5567,5580],{"className":5566},[342],[63,5568,5570],{"className":5569},[346],[348,5571,5572],{"xmlns":350},[352,5573,5574,5578],{},[355,5575,5576],{},[371,5577,612],{},[378,5579,612],{"encoding":380},[63,5581,5583],{"className":5582,"ariaHidden":386},[385],[63,5584,5586,5589],{"className":5585},[390],[63,5587],{"className":5588,"style":624},[394],[63,5590,612],{"className":5591},[399,472]," 的序列、每个 token 用 ",[63,5594,5596,5613],{"className":5595},[342],[63,5597,5599],{"className":5598},[346],[348,5600,5601],{"xmlns":350},[352,5602,5603,5611],{},[355,5604,5605],{},[368,5606,5607,5609],{},[371,5608,373],{},[3343,5610,3836],{},[378,5612,5353],{"encoding":380},[63,5614,5616],{"className":5615,"ariaHidden":386},[385],[63,5617,5619,5622],{"className":5618},[390],[63,5620],{"className":5621,"style":1616},[394],[63,5623,5625,5628],{"className":5624},[399],[63,5626,373],{"className":5627},[399,472],[63,5629,5631],{"className":5630},[476],[63,5632,5634,5660],{"className":5633},[411,412],[63,5635,5637,5657],{"className":5636},[416],[63,5638,5640],{"className":5639,"style":1532},[420],[63,5641,5642,5645],{"style":1535},[63,5643],{"className":5644,"style":817},[428],[63,5646,5648],{"className":5647},[433,434,435,436],[63,5649,5651],{"className":5650},[399,436],[63,5652,5654],{"className":5653},[399,5395,436],[63,5655,3836],{"className":5656},[399,436],[63,5658,507],{"className":5659},[506],[63,5661,5663],{"className":5662},[416],[63,5664,5666],{"className":5665,"style":1554},[420],[63,5667],{}," 维向量表示。位置编码使用相同形状的矩阵 ",[63,5670,5672,5710],{"className":5671},[342],[63,5673,5675],{"className":5674},[346],[348,5676,5677],{"xmlns":350},[352,5678,5679,5707],{},[355,5680,5681,5689,5691],{},[355,5682,5683,5686],{},[371,5684,5685],{"mathvariant":5431},"P",[371,5687,5688],{"mathvariant":5431},"E",[738,5690,740],{},[742,5692,5693,5695],{},[371,5694,747],{"mathvariant":746},[355,5696,5697,5699,5701],{},[371,5698,612],{},[738,5700,754],{},[368,5702,5703,5705],{},[371,5704,373],{},[3343,5706,3836],{},[378,5708,5709],{"encoding":380},"\\mathbf{PE} \\in \\mathbb{R}^{n \\times d_{\\text{model}}}",[63,5711,5713,5735],{"className":5712,"ariaHidden":386},[385],[63,5714,5716,5719,5726,5729,5732],{"className":5715},[390],[63,5717],{"className":5718,"style":5462},[394],[63,5720,5722],{"className":5721},[399],[63,5723,5725],{"className":5724},[399,5466],"PE",[63,5727],{"className":5728,"style":777},[776],[63,5730,740],{"className":5731},[781],[63,5733],{"className":5734,"style":777},[776],[63,5736,5738,5741],{"className":5737},[390],[63,5739],{"className":5740,"style":791},[394],[63,5742,5744,5747],{"className":5743},[399],[63,5745,747],{"className":5746},[399,798],[63,5748,5750],{"className":5749},[476],[63,5751,5753],{"className":5752},[411],[63,5754,5756],{"className":5755},[416],[63,5757,5759],{"className":5758,"style":791},[420],[63,5760,5761,5764],{"style":813},[63,5762],{"className":5763,"style":817},[428],[63,5765,5767],{"className":5766},[433,434,435,436],[63,5768,5770,5773,5776],{"className":5769},[399,436],[63,5771,612],{"className":5772},[399,472,436],[63,5774,754],{"className":5775},[830,436],[63,5777,5779,5782],{"className":5778},[399,436],[63,5780,373],{"className":5781},[399,472,436],[63,5783,5785],{"className":5784},[476],[63,5786,5788,5814],{"className":5787},[411,412],[63,5789,5791,5811],{"className":5790},[416],[63,5792,5794],{"className":5793,"style":486},[420],[63,5795,5796,5799],{"style":489},[63,5797],{"className":5798,"style":493},[428],[63,5800,5802],{"className":5801},[433,497,498,436],[63,5803,5805],{"className":5804},[399,436],[63,5806,5808],{"className":5807},[399,5395,436],[63,5809,3836],{"className":5810},[399,436],[63,5812,507],{"className":5813},[506],[63,5815,5817],{"className":5816},[416],[63,5818,5820],{"className":5819,"style":514},[420],[63,5821],{},"，最终输出为 ",[63,5824,5826,5849],{"className":5825},[342],[63,5827,5829],{"className":5828},[346],[348,5830,5831],{"xmlns":350},[352,5832,5833,5846],{},[355,5834,5835,5837,5840],{},[371,5836,2487],{"mathvariant":5431},[738,5838,5839],{},"+",[355,5841,5842,5844],{},[371,5843,5685],{"mathvariant":5431},[371,5845,5688],{"mathvariant":5431},[378,5847,5848],{"encoding":380},"\\mathbf{X} + \\mathbf{PE}",[63,5850,5852,5871],{"className":5851,"ariaHidden":386},[385],[63,5853,5855,5859,5862,5865,5868],{"className":5854},[390],[63,5856],{"className":5857,"style":5858},[394],"height:0.7694em;vertical-align:-0.0833em;",[63,5860,2487],{"className":5861},[399,5466],[63,5863],{"className":5864,"style":1920},[776],[63,5866,5839],{"className":5867},[830],[63,5869],{"className":5870,"style":1920},[776],[63,5872,5874,5878],{"className":5873},[390],[63,5875],{"className":5876,"style":5877},[394],"height:0.6861em;",[63,5879,5881],{"className":5880},[399],[63,5882,5725],{"className":5883},[399,5466],"。其中第 ",[63,5886,5888,5901],{"className":5887},[342],[63,5889,5891],{"className":5890},[346],[348,5892,5893],{"xmlns":350},[352,5894,5895,5899],{},[355,5896,5897],{},[371,5898,1069],{},[378,5900,1069],{"encoding":380},[63,5902,5904],{"className":5903,"ariaHidden":386},[385],[63,5905,5907,5911],{"className":5906},[390],[63,5908],{"className":5909,"style":5910},[394],"height:0.6595em;",[63,5912,1069],{"className":5913},[399,472]," 行、第 ",[63,5916,5918,5935],{"className":5917},[342],[63,5919,5921],{"className":5920},[346],[348,5922,5923],{"xmlns":350},[352,5924,5925,5932],{},[355,5926,5927,5929],{},[361,5928,2883],{},[371,5930,5931],{},"j",[378,5933,5934],{"encoding":380},"2j",[63,5936,5938],{"className":5937,"ariaHidden":386},[385],[63,5939,5941,5945,5948],{"className":5940},[390],[63,5942],{"className":5943,"style":5944},[394],"height:0.854em;vertical-align:-0.1944em;",[63,5946,2883],{"className":5947},[399],[63,5949,5931],{"className":5950,"style":5951},[399,472],"margin-right:0.05724em;",[63,5953,5955,5975],{"className":5954},[342],[63,5956,5958],{"className":5957},[346],[348,5959,5960],{"xmlns":350},[352,5961,5962,5972],{},[355,5963,5964,5966,5968,5970],{},[361,5965,2883],{},[371,5967,5931],{},[738,5969,5839],{},[361,5971,363],{},[378,5973,5974],{"encoding":380},"2j+1",[63,5976,5978,5999],{"className":5977,"ariaHidden":386},[385],[63,5979,5981,5984,5987,5990,5993,5996],{"className":5980},[390],[63,5982],{"className":5983,"style":5944},[394],[63,5985,2883],{"className":5986},[399],[63,5988,5931],{"className":5989,"style":5951},[399,472],[63,5991],{"className":5992,"style":1920},[776],[63,5994,5839],{"className":5995},[830],[63,5997],{"className":5998,"style":1920},[776],[63,6000,6002,6005],{"className":6001},[390],[63,6003],{"className":6004,"style":1814},[394],[63,6006,363],{"className":6007},[399]," 列上的元素分别为：",[14,6010,6011],{},[63,6012,6014,6082],{"className":6013},[342],[63,6015,6017],{"className":6016},[346],[348,6018,6019],{"xmlns":350},[352,6020,6021,6079],{},[355,6022,6023,6025,6043,6045,6048,6050],{},[371,6024,5685],{},[368,6026,6027,6029],{},[371,6028,5688],{},[355,6030,6031,6033,6035,6037,6039,6041],{},[738,6032,1078],{"stretchy":1077},[371,6034,1069],{},[738,6036,1083],{"separator":386},[361,6038,2883],{},[371,6040,5931],{},[738,6042,1092],{"stretchy":1077},[738,6044,204],{},[371,6046,6047],{},"sin",[738,6049,1100],{},[355,6051,6052,6054,6077],{},[738,6053,1078],{"fence":386},[358,6055,6056,6058],{},[371,6057,1069],{},[742,6059,6060,6063],{},[361,6061,6062],{},"10000",[355,6064,6065,6067,6069,6071],{},[361,6066,2883],{},[371,6068,5931],{},[371,6070,2889],{"mathvariant":1053},[368,6072,6073,6075],{},[371,6074,373],{},[3343,6076,3836],{},[738,6078,1092],{"fence":386},[378,6080,6081],{"encoding":380}," PE_{(i,2j)} = \\sin\\left(\\frac{i}{10000^{2j/d_{\\text{model}}}}\\right)",[63,6083,6085,6165],{"className":6084,"ariaHidden":386},[385],[63,6086,6088,6092,6095,6156,6159,6162],{"className":6087},[390],[63,6089],{"className":6090,"style":6091},[394],"height:1.0385em;vertical-align:-0.3552em;",[63,6093,5685],{"className":6094,"style":3251},[399,472],[63,6096,6098,6102],{"className":6097},[399],[63,6099,5688],{"className":6100,"style":6101},[399,472],"margin-right:0.05764em;",[63,6103,6105],{"className":6104},[476],[63,6106,6108,6147],{"className":6107},[411,412],[63,6109,6111,6144],{"className":6110},[416],[63,6112,6114],{"className":6113,"style":486},[420],[63,6115,6117,6120],{"style":6116},"top:-2.5198em;margin-left:-0.0576em;margin-right:0.05em;",[63,6118],{"className":6119,"style":817},[428],[63,6121,6123],{"className":6122},[433,434,435,436],[63,6124,6126,6129,6132,6135,6138,6141],{"className":6125},[399,436],[63,6127,1078],{"className":6128},[403,436],[63,6130,1069],{"className":6131},[399,472,436],[63,6133,1083],{"className":6134},[1173,436],[63,6136,2883],{"className":6137},[399,436],[63,6139,5931],{"className":6140,"style":5951},[399,472,436],[63,6142,1092],{"className":6143},[593,436],[63,6145,507],{"className":6146},[506],[63,6148,6150],{"className":6149},[416],[63,6151,6154],{"className":6152,"style":6153},[420],"height:0.3552em;",[63,6155],{},[63,6157],{"className":6158,"style":777},[776],[63,6160,204],{"className":6161},[781],[63,6163],{"className":6164,"style":777},[776],[63,6166,6168,6171,6174,6177],{"className":6167},[390],[63,6169],{"className":6170,"style":1208},[394],[63,6172,6047],{"className":6173},[1212],[63,6175],{"className":6176,"style":1177},[776],[63,6178,6180,6186,6345],{"className":6179},[1222],[63,6181,6183],{"className":6182,"style":1227},[403,1226],[63,6184,1078],{"className":6185},[1231,1232],[63,6187,6189,6192,6342],{"className":6188},[399],[63,6190],{"className":6191},[403,404],[63,6193,6195],{"className":6194},[358],[63,6196,6198,6333],{"className":6197},[411,412],[63,6199,6201,6330],{"className":6200},[416],[63,6202,6205,6308,6316],{"className":6203,"style":6204},[420],"height:0.8557em;",[63,6206,6208,6211],{"style":6207},"top:-2.5648em;",[63,6209],{"className":6210,"style":429},[428],[63,6212,6214],{"className":6213},[433,434,435,436],[63,6215,6217,6221],{"className":6216},[399,436],[63,6218,6220],{"className":6219},[399,436],"1000",[63,6222,6224,6227],{"className":6223},[399,436],[63,6225,1802],{"className":6226},[399,436],[63,6228,6230],{"className":6229},[476],[63,6231,6233],{"className":6232},[411],[63,6234,6236],{"className":6235},[416],[63,6237,6240],{"className":6238,"style":6239},[420],"height:0.8932em;",[63,6241,6243,6247],{"style":6242},"top:-2.8932em;margin-right:0.0714em;",[63,6244],{"className":6245,"style":6246},[428],"height:2.5357em;",[63,6248,6250],{"className":6249},[433,497,498,436],[63,6251,6253,6256,6259,6262],{"className":6252},[399,436],[63,6254,2883],{"className":6255},[399,436],[63,6257,5931],{"className":6258,"style":5951},[399,472,436],[63,6260,2889],{"className":6261},[399,436],[63,6263,6265,6268],{"className":6264},[399,436],[63,6266,373],{"className":6267},[399,472,436],[63,6269,6271],{"className":6270},[476],[63,6272,6274,6299],{"className":6273},[411,412],[63,6275,6277,6296],{"className":6276},[416],[63,6278,6280],{"className":6279,"style":486},[420],[63,6281,6283,6287],{"style":6282},"top:-2.3448em;margin-left:0em;margin-right:0.1em;",[63,6284],{"className":6285,"style":6286},[428],"height:2.6944em;",[63,6288,6290],{"className":6289},[399,436],[63,6291,6293],{"className":6292},[399,5395,436],[63,6294,3836],{"className":6295},[399,436],[63,6297,507],{"className":6298},[506],[63,6300,6302],{"className":6301},[416],[63,6303,6306],{"className":6304,"style":6305},[420],"height:0.3496em;",[63,6307],{},[63,6309,6310,6313],{"style":554},[63,6311],{"className":6312,"style":429},[428],[63,6314],{"className":6315,"style":562},[561],[63,6317,6318,6321],{"style":565},[63,6319],{"className":6320,"style":429},[428],[63,6322,6324],{"className":6323},[433,434,435,436],[63,6325,6327],{"className":6326},[399,436],[63,6328,1069],{"className":6329},[399,472,436],[63,6331,507],{"className":6332},[506],[63,6334,6336],{"className":6335},[416],[63,6337,6340],{"className":6338,"style":6339},[420],"height:0.4352em;",[63,6341],{},[63,6343],{"className":6344},[593,404],[63,6346,6348],{"className":6347,"style":1227},[593,1226],[63,6349,1092],{"className":6350},[1231,1232],[14,6352,6353],{},[63,6354,6356,6427],{"className":6355},[342],[63,6357,6359],{"className":6358},[346],[348,6360,6361],{"xmlns":350},[352,6362,6363,6424],{},[355,6364,6365,6367,6389,6391,6394,6396],{},[371,6366,5685],{},[368,6368,6369,6371],{},[371,6370,5688],{},[355,6372,6373,6375,6377,6379,6381,6383,6385,6387],{},[738,6374,1078],{"stretchy":1077},[371,6376,1069],{},[738,6378,1083],{"separator":386},[361,6380,2883],{},[371,6382,5931],{},[738,6384,5839],{},[361,6386,363],{},[738,6388,1092],{"stretchy":1077},[738,6390,204],{},[371,6392,6393],{},"cos",[738,6395,1100],{},[355,6397,6398,6400,6422],{},[738,6399,1078],{"fence":386},[358,6401,6402,6404],{},[371,6403,1069],{},[742,6405,6406,6408],{},[361,6407,6062],{},[355,6409,6410,6412,6414,6416],{},[361,6411,2883],{},[371,6413,5931],{},[371,6415,2889],{"mathvariant":1053},[368,6417,6418,6420],{},[371,6419,373],{},[3343,6421,3836],{},[738,6423,1092],{"fence":386},[378,6425,6426],{"encoding":380}," PE_{(i,2j+1)} = \\cos\\left(\\frac{i}{10000^{2j/d_{\\text{model}}}}\\right)",[63,6428,6430,6512],{"className":6429,"ariaHidden":386},[385],[63,6431,6433,6436,6439,6503,6506,6509],{"className":6432},[390],[63,6434],{"className":6435,"style":6091},[394],[63,6437,5685],{"className":6438,"style":3251},[399,472],[63,6440,6442,6445],{"className":6441},[399],[63,6443,5688],{"className":6444,"style":6101},[399,472],[63,6446,6448],{"className":6447},[476],[63,6449,6451,6495],{"className":6450},[411,412],[63,6452,6454,6492],{"className":6453},[416],[63,6455,6457],{"className":6456,"style":486},[420],[63,6458,6459,6462],{"style":6116},[63,6460],{"className":6461,"style":817},[428],[63,6463,6465],{"className":6464},[433,434,435,436],[63,6466,6468,6471,6474,6477,6480,6483,6486,6489],{"className":6467},[399,436],[63,6469,1078],{"className":6470},[403,436],[63,6472,1069],{"className":6473},[399,472,436],[63,6475,1083],{"className":6476},[1173,436],[63,6478,2883],{"className":6479},[399,436],[63,6481,5931],{"className":6482,"style":5951},[399,472,436],[63,6484,5839],{"className":6485},[830,436],[63,6487,363],{"className":6488},[399,436],[63,6490,1092],{"className":6491},[593,436],[63,6493,507],{"className":6494},[506],[63,6496,6498],{"className":6497},[416],[63,6499,6501],{"className":6500,"style":6153},[420],[63,6502],{},[63,6504],{"className":6505,"style":777},[776],[63,6507,204],{"className":6508},[781],[63,6510],{"className":6511,"style":777},[776],[63,6513,6515,6518,6521,6524],{"className":6514},[390],[63,6516],{"className":6517,"style":1208},[394],[63,6519,6393],{"className":6520},[1212],[63,6522],{"className":6523,"style":1177},[776],[63,6525,6527,6533,6682],{"className":6526},[1222],[63,6528,6530],{"className":6529,"style":1227},[403,1226],[63,6531,1078],{"className":6532},[1231,1232],[63,6534,6536,6539,6679],{"className":6535},[399],[63,6537],{"className":6538},[403,404],[63,6540,6542],{"className":6541},[358],[63,6543,6545,6671],{"className":6544},[411,412],[63,6546,6548,6668],{"className":6547},[416],[63,6549,6551,6646,6654],{"className":6550,"style":6204},[420],[63,6552,6553,6556],{"style":6207},[63,6554],{"className":6555,"style":429},[428],[63,6557,6559],{"className":6558},[433,434,435,436],[63,6560,6562,6565],{"className":6561},[399,436],[63,6563,6220],{"className":6564},[399,436],[63,6566,6568,6571],{"className":6567},[399,436],[63,6569,1802],{"className":6570},[399,436],[63,6572,6574],{"className":6573},[476],[63,6575,6577],{"className":6576},[411],[63,6578,6580],{"className":6579},[416],[63,6581,6583],{"className":6582,"style":6239},[420],[63,6584,6585,6588],{"style":6242},[63,6586],{"className":6587,"style":6246},[428],[63,6589,6591],{"className":6590},[433,497,498,436],[63,6592,6594,6597,6600,6603],{"className":6593},[399,436],[63,6595,2883],{"className":6596},[399,436],[63,6598,5931],{"className":6599,"style":5951},[399,472,436],[63,6601,2889],{"className":6602},[399,436],[63,6604,6606,6609],{"className":6605},[399,436],[63,6607,373],{"className":6608},[399,472,436],[63,6610,6612],{"className":6611},[476],[63,6613,6615,6638],{"className":6614},[411,412],[63,6616,6618,6635],{"className":6617},[416],[63,6619,6621],{"className":6620,"style":486},[420],[63,6622,6623,6626],{"style":6282},[63,6624],{"className":6625,"style":6286},[428],[63,6627,6629],{"className":6628},[399,436],[63,6630,6632],{"className":6631},[399,5395,436],[63,6633,3836],{"className":6634},[399,436],[63,6636,507],{"className":6637},[506],[63,6639,6641],{"className":6640},[416],[63,6642,6644],{"className":6643,"style":6305},[420],[63,6645],{},[63,6647,6648,6651],{"style":554},[63,6649],{"className":6650,"style":429},[428],[63,6652],{"className":6653,"style":562},[561],[63,6655,6656,6659],{"style":565},[63,6657],{"className":6658,"style":429},[428],[63,6660,6662],{"className":6661},[433,434,435,436],[63,6663,6665],{"className":6664},[399,436],[63,6666,1069],{"className":6667},[399,472,436],[63,6669,507],{"className":6670},[506],[63,6672,6674],{"className":6673},[416],[63,6675,6677],{"className":6676,"style":6339},[420],[63,6678],{},[63,6680],{"className":6681},[593,404],[63,6683,6685],{"className":6684,"style":1227},[593,1226],[63,6686,1092],{"className":6687},[1231,1232],[14,6689,3279,6690,6718,6719,6747,6748,6783,6784,6840,6841,147,6869,6962,6963,7039],{},[63,6691,6693,6706],{"className":6692},[342],[63,6694,6696],{"className":6695},[346],[348,6697,6698],{"xmlns":350},[352,6699,6700,6704],{},[355,6701,6702],{},[371,6703,1069],{},[378,6705,1069],{"encoding":380},[63,6707,6709],{"className":6708,"ariaHidden":386},[385],[63,6710,6712,6715],{"className":6711},[390],[63,6713],{"className":6714,"style":5910},[394],[63,6716,1069],{"className":6717},[399,472]," 表示位置，",[63,6720,6722,6735],{"className":6721},[342],[63,6723,6725],{"className":6724},[346],[348,6726,6727],{"xmlns":350},[352,6728,6729,6733],{},[355,6730,6731],{},[371,6732,5931],{},[378,6734,5931],{"encoding":380},[63,6736,6738],{"className":6737,"ariaHidden":386},[385],[63,6739,6741,6744],{"className":6740},[390],[63,6742],{"className":6743,"style":5944},[394],[63,6745,5931],{"className":6746,"style":5951},[399,472]," 表示维度索引。也就是说，位置编码的每一维都对应一条不同频率的正弦/余弦曲线，波长从 ",[63,6749,6751,6768],{"className":6750},[342],[63,6752,6754],{"className":6753},[346],[348,6755,6756],{"xmlns":350},[352,6757,6758,6765],{},[355,6759,6760,6762],{},[361,6761,2883],{},[371,6763,6764],{},"π",[378,6766,6767],{"encoding":380},"2\\pi",[63,6769,6771],{"className":6770,"ariaHidden":386},[385],[63,6772,6774,6777,6780],{"className":6773},[390],[63,6775],{"className":6776,"style":1814},[394],[63,6778,2883],{"className":6779},[399],[63,6781,6764],{"className":6782,"style":718},[399,472]," 到 ",[63,6785,6787,6807],{"className":6786},[342],[63,6788,6790],{"className":6789},[346],[348,6791,6792],{"xmlns":350},[352,6793,6794,6804],{},[355,6795,6796,6798,6800,6802],{},[361,6797,6062],{},[738,6799,1864],{},[361,6801,2883],{},[371,6803,6764],{},[378,6805,6806],{"encoding":380},"10000 \\cdot 2\\pi",[63,6808,6810,6828],{"className":6809,"ariaHidden":386},[385],[63,6811,6813,6816,6819,6822,6825],{"className":6812},[390],[63,6814],{"className":6815,"style":1814},[394],[63,6817,6062],{"className":6818},[399],[63,6820],{"className":6821,"style":1920},[776],[63,6823,1864],{"className":6824},[830],[63,6826],{"className":6827,"style":1920},[776],[63,6829,6831,6834,6837],{"className":6830},[390],[63,6832],{"className":6833,"style":1814},[394],[63,6835,2883],{"className":6836},[399],[63,6838,6764],{"className":6839,"style":718},[399,472]," 按几何级数增长。选择这一形式的直觉在于：对于任意固定偏移 ",[63,6842,6844,6857],{"className":6843},[342],[63,6845,6847],{"className":6846},[346],[348,6848,6849],{"xmlns":350},[352,6850,6851,6855],{},[355,6852,6853],{},[371,6854,376],{},[378,6856,376],{"encoding":380},[63,6858,6860],{"className":6859,"ariaHidden":386},[385],[63,6861,6863,6866],{"className":6862},[390],[63,6864],{"className":6865,"style":684},[394],[63,6867,376],{"className":6868,"style":502},[399,472],[63,6870,6872,6898],{"className":6871},[342],[63,6873,6875],{"className":6874},[346],[348,6876,6877],{"xmlns":350},[352,6878,6879,6895],{},[355,6880,6881,6883],{},[371,6882,5685],{},[368,6884,6885,6887],{},[371,6886,5688],{},[355,6888,6889,6891,6893],{},[371,6890,1069],{},[738,6892,5839],{},[371,6894,376],{},[378,6896,6897],{"encoding":380},"PE_{i+k}",[63,6899,6901],{"className":6900,"ariaHidden":386},[385],[63,6902,6904,6908,6911],{"className":6903},[390],[63,6905],{"className":6906,"style":6907},[394],"height:0.8917em;vertical-align:-0.2083em;",[63,6909,5685],{"className":6910,"style":3251},[399,472],[63,6912,6914,6917],{"className":6913},[399],[63,6915,5688],{"className":6916,"style":6101},[399,472],[63,6918,6920],{"className":6919},[476],[63,6921,6923,6953],{"className":6922},[411,412],[63,6924,6926,6950],{"className":6925},[416],[63,6927,6929],{"className":6928,"style":1532},[420],[63,6930,6932,6935],{"style":6931},"top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;",[63,6933],{"className":6934,"style":817},[428],[63,6936,6938],{"className":6937},[433,434,435,436],[63,6939,6941,6944,6947],{"className":6940},[399,436],[63,6942,1069],{"className":6943},[399,472,436],[63,6945,5839],{"className":6946},[830,436],[63,6948,376],{"className":6949,"style":502},[399,472,436],[63,6951,507],{"className":6952},[506],[63,6954,6956],{"className":6955},[416],[63,6957,6960],{"className":6958,"style":6959},[420],"height:0.2083em;",[63,6961],{}," 都可以表示为 ",[63,6964,6966,6986],{"className":6965},[342],[63,6967,6969],{"className":6968},[346],[348,6970,6971],{"xmlns":350},[352,6972,6973,6983],{},[355,6974,6975,6977],{},[371,6976,5685],{},[368,6978,6979,6981],{},[371,6980,5688],{},[371,6982,1069],{},[378,6984,6985],{"encoding":380},"PE_i",[63,6987,6989],{"className":6988,"ariaHidden":386},[385],[63,6990,6992,6996,6999],{"className":6991},[390],[63,6993],{"className":6994,"style":6995},[394],"height:0.8333em;vertical-align:-0.15em;",[63,6997,5685],{"className":6998,"style":3251},[399,472],[63,7000,7002,7005],{"className":7001},[399],[63,7003,5688],{"className":7004,"style":6101},[399,472],[63,7006,7008],{"className":7007},[476],[63,7009,7011,7031],{"className":7010},[411,412],[63,7012,7014,7028],{"className":7013},[416],[63,7015,7017],{"className":7016,"style":2080},[420],[63,7018,7019,7022],{"style":6931},[63,7020],{"className":7021,"style":817},[428],[63,7023,7025],{"className":7024},[433,434,435,436],[63,7026,1069],{"className":7027},[399,472,436],[63,7029,507],{"className":7030},[506],[63,7032,7034],{"className":7033},[416],[63,7035,7037],{"className":7036,"style":1554},[420],[63,7038],{}," 的线性变换，从而让模型更容易学到\"相对位置\"这种关系。",[54,7041,7043],{"className":2468,"code":7042,"language":2470,"meta":59,"style":59},"class PositionalEncoding(nn.Block):\n    \"\"\"位置编码\"\"\"\n    def __init__(self, num_hiddens, dropout, max_len=1000):\n        super(PositionalEncoding, self).__init__()\n        self.dropout = nn.Dropout(dropout)\n        # 创建一个足够长的P\n        self.P = np.zeros((1, max_len, num_hiddens))\n        X = np.arange(max_len).reshape(-1, 1) / np.power(\n            10000, np.arange(0, num_hiddens, 2) / num_hiddens)\n        self.P[:, :, 0::2] = np.sin(X)\n        self.P[:, :, 1::2] = np.cos(X)\n\n    def forward(self, X):\n        X = X + self.P[:, :X.shape[1], :].as_in_ctx(X.ctx)\n        return self.dropout(X)\n",[49,7044,7045,7063,7068,7097,7113,7123,7128,7145,7170,7192,7224,7251,7255,7271,7304],{"__ignoreMap":59},[63,7046,7047,7049,7052,7054,7056,7058,7061],{"class":65,"line":66},[63,7048,2695],{"class":196},[63,7050,7051],{"class":2698}," PositionalEncoding",[63,7053,1078],{"class":2483},[63,7055,2705],{"class":2704},[63,7057,1142],{"class":2483},[63,7059,7060],{"class":2704},"Block",[63,7062,2496],{"class":2483},[63,7064,7065],{"class":65,"line":88},[63,7066,7067],{"class":2501},"    \"\"\"位置编码\"\"\"\n",[63,7069,7070,7072,7074,7076,7078,7080,7082,7084,7086,7088,7091,7093,7095],{"class":65,"line":95},[63,7071,2722],{"class":196},[63,7073,2725],{"class":73},[63,7075,1078],{"class":2483},[63,7077,2731],{"class":2730},[63,7079,2490],{"class":2483},[63,7081,4929],{"class":2486},[63,7083,2490],{"class":2483},[63,7085,2736],{"class":2486},[63,7087,2490],{"class":2483},[63,7089,7090],{"class":2486},"max_len",[63,7092,204],{"class":196},[63,7094,6220],{"class":73},[63,7096,2496],{"class":2483},[63,7098,7099,7101,7104,7106,7108,7110],{"class":65,"line":108},[63,7100,2752],{"class":2751},[63,7102,7103],{"class":2483},"(PositionalEncoding, ",[63,7105,2731],{"class":81},[63,7107,2760],{"class":2483},[63,7109,2763],{"class":73},[63,7111,7112],{"class":2483},"()\n",[63,7114,7115,7117,7119,7121],{"class":65,"line":113},[63,7116,2775],{"class":81},[63,7118,2778],{"class":2483},[63,7120,204],{"class":196},[63,7122,2783],{"class":2483},[63,7124,7125],{"class":65,"line":2548},[63,7126,7127],{"class":2507},"        # 创建一个足够长的P\n",[63,7129,7130,7132,7135,7137,7140,7142],{"class":65,"line":2556},[63,7131,2775],{"class":81},[63,7133,7134],{"class":2483},".P ",[63,7136,204],{"class":196},[63,7138,7139],{"class":2483}," np.zeros((",[63,7141,363],{"class":73},[63,7143,7144],{"class":2483},", max_len, num_hiddens))\n",[63,7146,7147,7149,7151,7154,7156,7158,7160,7162,7165,7167],{"class":65,"line":2567},[63,7148,2634],{"class":2483},[63,7150,204],{"class":196},[63,7152,7153],{"class":2483}," np.arange(max_len).reshape(",[63,7155,2618],{"class":196},[63,7157,363],{"class":73},[63,7159,2490],{"class":2483},[63,7161,363],{"class":73},[63,7163,7164],{"class":2483},") ",[63,7166,2889],{"class":196},[63,7168,7169],{"class":2483}," np.power(\n",[63,7171,7172,7175,7178,7180,7183,7185,7187,7189],{"class":65,"line":2584},[63,7173,7174],{"class":73},"            10000",[63,7176,7177],{"class":2483},", np.arange(",[63,7179,1802],{"class":73},[63,7181,7182],{"class":2483},", num_hiddens, ",[63,7184,2883],{"class":73},[63,7186,7164],{"class":2483},[63,7188,2889],{"class":196},[63,7190,7191],{"class":2483}," num_hiddens)\n",[63,7193,7194,7196,7199,7203,7205,7207,7209,7211,7214,7216,7219,7221],{"class":65,"line":2600},[63,7195,2775],{"class":81},[63,7197,7198],{"class":2483},".P[",[63,7200,7202],{"class":7201},"sDoOe",":",[63,7204,2490],{"class":2483},[63,7206,7202],{"class":7201},[63,7208,2490],{"class":2483},[63,7210,1802],{"class":73},[63,7212,7213],{"class":7201},"::",[63,7215,2883],{"class":73},[63,7217,7218],{"class":2483},"] ",[63,7220,204],{"class":196},[63,7222,7223],{"class":2483}," np.sin(X)\n",[63,7225,7226,7228,7230,7232,7234,7236,7238,7240,7242,7244,7246,7248],{"class":65,"line":2608},[63,7227,2775],{"class":81},[63,7229,7198],{"class":2483},[63,7231,7202],{"class":7201},[63,7233,2490],{"class":2483},[63,7235,7202],{"class":7201},[63,7237,2490],{"class":2483},[63,7239,363],{"class":73},[63,7241,7213],{"class":7201},[63,7243,2883],{"class":73},[63,7245,7218],{"class":2483},[63,7247,204],{"class":196},[63,7249,7250],{"class":2483}," np.cos(X)\n",[63,7252,7253],{"class":65,"line":2625},[63,7254,92],{"emptyLinePlaceholder":91},[63,7256,7257,7259,7261,7263,7265,7267,7269],{"class":65,"line":2631},[63,7258,2722],{"class":196},[63,7260,2814],{"class":69},[63,7262,1078],{"class":2483},[63,7264,2731],{"class":2730},[63,7266,2490],{"class":2483},[63,7268,2487],{"class":2486},[63,7270,2496],{"class":2483},[63,7272,7273,7275,7277,7280,7282,7284,7286,7288,7290,7292,7295,7297,7299,7301],{"class":65,"line":2656},[63,7274,2634],{"class":2483},[63,7276,204],{"class":196},[63,7278,7279],{"class":2483}," X ",[63,7281,5839],{"class":196},[63,7283,5287],{"class":81},[63,7285,7198],{"class":2483},[63,7287,7202],{"class":7201},[63,7289,2490],{"class":2483},[63,7291,7202],{"class":7201},[63,7293,7294],{"class":2483},"X.shape[",[63,7296,363],{"class":73},[63,7298,4861],{"class":2483},[63,7300,7202],{"class":7201},[63,7302,7303],{"class":2483},"].as_in_ctx(X.ctx)\n",[63,7305,7306,7308,7310],{"class":65,"line":2669},[63,7307,2530],{"class":196},[63,7309,5287],{"class":81},[63,7311,7312],{"class":2483},".dropout(X)\n",[10,7314,7315],{"id":7315},"前馈神经网络",[14,7317,7318],{},"除了注意力子层，编码器与解码器的每一层都包含一个 position-wise 的全连接前馈网络（FFN）：它对序列中每个位置独立、且共享权重地施加同一组变换，包含两个线性层与中间的 ReLU 激活：",[14,7320,7321],{},[63,7322,7324,7402],{"className":7323},[342],[63,7325,7327],{"className":7326},[346],[348,7328,7329],{"xmlns":350},[352,7330,7331,7399],{},[355,7332,7333,7343,7345,7348,7350,7352,7355,7357,7359,7361,7363,7366,7368,7374,7376,7383,7385,7391,7393],{},[355,7334,7335,7338,7340],{},[371,7336,7337],{"mathvariant":1053},"F",[371,7339,7337],{"mathvariant":1053},[371,7341,7342],{"mathvariant":1053},"N",[738,7344,1078],{"stretchy":1077},[371,7346,7347],{},"x",[738,7349,1092],{"stretchy":1077},[738,7351,204],{},[371,7353,7354],{},"max",[738,7356,1100],{},[738,7358,1078],{"stretchy":1077},[361,7360,1802],{},[738,7362,1083],{"separator":386},[3343,7364,7365],{}," ",[371,7367,7347],{},[368,7369,7370,7372],{},[371,7371,3059],{},[361,7373,363],{},[738,7375,5839],{},[368,7377,7378,7381],{},[371,7379,7380],{},"b",[361,7382,363],{},[738,7384,1092],{"stretchy":1077},[368,7386,7387,7389],{},[371,7388,3059],{},[361,7390,2883],{},[738,7392,5839],{},[368,7394,7395,7397],{},[371,7396,7380],{},[361,7398,2883],{},[378,7400,7401],{"encoding":380}," \\mathrm{FFN}(x) = \\max(0,\\,xW_1+b_1)W_2+b_2",[63,7403,7405,7436,7513,7611],{"className":7404,"ariaHidden":386},[385],[63,7406,7408,7411,7418,7421,7424,7427,7430,7433],{"className":7407},[390],[63,7409],{"className":7410,"style":1155},[394],[63,7412,7414],{"className":7413},[399],[63,7415,7417],{"className":7416},[399,1162],"FFN",[63,7419,1078],{"className":7420},[403],[63,7422,7347],{"className":7423},[399,472],[63,7425,1092],{"className":7426},[593],[63,7428],{"className":7429,"style":777},[776],[63,7431,204],{"className":7432},[781],[63,7434],{"className":7435,"style":777},[776],[63,7437,7439,7442,7445,7448,7451,7454,7457,7460,7463,7504,7507,7510],{"className":7438},[390],[63,7440],{"className":7441,"style":1155},[394],[63,7443,7354],{"className":7444},[1212],[63,7446,1078],{"className":7447},[403],[63,7449,1802],{"className":7450},[399],[63,7452,1083],{"className":7453},[1173],[63,7455],{"className":7456,"style":1177},[776],[63,7458],{"className":7459,"style":1177},[776],[63,7461,7347],{"className":7462},[399,472],[63,7464,7466,7469],{"className":7465},[399],[63,7467,3059],{"className":7468,"style":3251},[399,472],[63,7470,7472],{"className":7471},[476],[63,7473,7475,7496],{"className":7474},[411,412],[63,7476,7478,7493],{"className":7477},[416],[63,7479,7481],{"className":7480,"style":3157},[420],[63,7482,7484,7487],{"style":7483},"top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;",[63,7485],{"className":7486,"style":817},[428],[63,7488,7490],{"className":7489},[433,434,435,436],[63,7491,363],{"className":7492},[399,436],[63,7494,507],{"className":7495},[506],[63,7497,7499],{"className":7498},[416],[63,7500,7502],{"className":7501,"style":1554},[420],[63,7503],{},[63,7505],{"className":7506,"style":1920},[776],[63,7508,5839],{"className":7509},[830],[63,7511],{"className":7512,"style":1920},[776],[63,7514,7516,7519,7559,7562,7602,7605,7608],{"className":7515},[390],[63,7517],{"className":7518,"style":1155},[394],[63,7520,7522,7525],{"className":7521},[399],[63,7523,7380],{"className":7524},[399,472],[63,7526,7528],{"className":7527},[476],[63,7529,7531,7551],{"className":7530},[411,412],[63,7532,7534,7548],{"className":7533},[416],[63,7535,7537],{"className":7536,"style":3157},[420],[63,7538,7539,7542],{"style":1535},[63,7540],{"className":7541,"style":817},[428],[63,7543,7545],{"className":7544},[433,434,435,436],[63,7546,363],{"className":7547},[399,436],[63,7549,507],{"className":7550},[506],[63,7552,7554],{"className":7553},[416],[63,7555,7557],{"className":7556,"style":1554},[420],[63,7558],{},[63,7560,1092],{"className":7561},[593],[63,7563,7565,7568],{"className":7564},[399],[63,7566,3059],{"className":7567,"style":3251},[399,472],[63,7569,7571],{"className":7570},[476],[63,7572,7574,7594],{"className":7573},[411,412],[63,7575,7577,7591],{"className":7576},[416],[63,7578,7580],{"className":7579,"style":3157},[420],[63,7581,7582,7585],{"style":7483},[63,7583],{"className":7584,"style":817},[428],[63,7586,7588],{"className":7587},[433,434,435,436],[63,7589,2883],{"className":7590},[399,436],[63,7592,507],{"className":7593},[506],[63,7595,7597],{"className":7596},[416],[63,7598,7600],{"className":7599,"style":1554},[420],[63,7601],{},[63,7603],{"className":7604,"style":1920},[776],[63,7606,5839],{"className":7607},[830],[63,7609],{"className":7610,"style":1920},[776],[63,7612,7614,7617],{"className":7613},[390],[63,7615],{"className":7616,"style":1616},[394],[63,7618,7620,7623],{"className":7619},[399],[63,7621,7380],{"className":7622},[399,472],[63,7624,7626],{"className":7625},[476],[63,7627,7629,7649],{"className":7628},[411,412],[63,7630,7632,7646],{"className":7631},[416],[63,7633,7635],{"className":7634,"style":3157},[420],[63,7636,7637,7640],{"style":1535},[63,7638],{"className":7639,"style":817},[428],[63,7641,7643],{"className":7642},[433,434,435,436],[63,7644,2883],{"className":7645},[399,436],[63,7647,507],{"className":7648},[506],[63,7650,7652],{"className":7651},[416],[63,7653,7655],{"className":7654,"style":1554},[420],[63,7656],{},[14,7658,7659],{},"直观上，注意力层负责\"在序列内交换信息\"，而 FFN 负责\"对每个位置做非线性变换\"，二者交替堆叠，使模型既能整合上下文又能拟合复杂的特征映射。",[54,7661,7663],{"className":2468,"code":7662,"language":2470,"meta":59,"style":59},"class PositionWiseFFN(nn.Module):\n    \"\"\"基于位置的前馈网络\"\"\"\n    def __init__(self, ffn_num_input, ffn_num_hiddens, ffn_num_outputs,\n                 **kwargs):\n        super(PositionWiseFFN, self).__init__(**kwargs)\n        self.dense1 = nn.Linear(ffn_num_input, ffn_num_hiddens)\n        self.relu = nn.ReLU()\n        self.dense2 = nn.Linear(ffn_num_hiddens, ffn_num_outputs)\n\n    def forward(self, X):\n        return self.dense2(self.relu(self.dense1(X)))\n",[49,7664,7665,7682,7687,7714,7723,7742,7754,7766,7778,7782,7798],{"__ignoreMap":59},[63,7666,7667,7669,7672,7674,7676,7678,7680],{"class":65,"line":66},[63,7668,2695],{"class":196},[63,7670,7671],{"class":2698}," PositionWiseFFN",[63,7673,1078],{"class":2483},[63,7675,2705],{"class":2704},[63,7677,1142],{"class":2483},[63,7679,2710],{"class":2704},[63,7681,2496],{"class":2483},[63,7683,7684],{"class":65,"line":88},[63,7685,7686],{"class":2501},"    \"\"\"基于位置的前馈网络\"\"\"\n",[63,7688,7689,7691,7693,7695,7697,7699,7702,7704,7707,7709,7712],{"class":65,"line":95},[63,7690,2722],{"class":196},[63,7692,2725],{"class":73},[63,7694,1078],{"class":2483},[63,7696,2731],{"class":2730},[63,7698,2490],{"class":2483},[63,7700,7701],{"class":2486},"ffn_num_input",[63,7703,2490],{"class":2483},[63,7705,7706],{"class":2486},"ffn_num_hiddens",[63,7708,2490],{"class":2483},[63,7710,7711],{"class":2486},"ffn_num_outputs",[63,7713,3891],{"class":2483},[63,7715,7716,7719,7721],{"class":65,"line":108},[63,7717,7718],{"class":196},"                 **",[63,7720,2744],{"class":2486},[63,7722,2496],{"class":2483},[63,7724,7725,7727,7730,7732,7734,7736,7738,7740],{"class":65,"line":113},[63,7726,2752],{"class":2751},[63,7728,7729],{"class":2483},"(PositionWiseFFN, ",[63,7731,2731],{"class":81},[63,7733,2760],{"class":2483},[63,7735,2763],{"class":73},[63,7737,1078],{"class":2483},[63,7739,2741],{"class":196},[63,7741,2770],{"class":2483},[63,7743,7744,7746,7749,7751],{"class":65,"line":2548},[63,7745,2775],{"class":81},[63,7747,7748],{"class":2483},".dense1 ",[63,7750,204],{"class":196},[63,7752,7753],{"class":2483}," nn.Linear(ffn_num_input, ffn_num_hiddens)\n",[63,7755,7756,7758,7761,7763],{"class":65,"line":2556},[63,7757,2775],{"class":81},[63,7759,7760],{"class":2483},".relu ",[63,7762,204],{"class":196},[63,7764,7765],{"class":2483}," nn.ReLU()\n",[63,7767,7768,7770,7773,7775],{"class":65,"line":2567},[63,7769,2775],{"class":81},[63,7771,7772],{"class":2483},".dense2 ",[63,7774,204],{"class":196},[63,7776,7777],{"class":2483}," nn.Linear(ffn_num_hiddens, ffn_num_outputs)\n",[63,7779,7780],{"class":65,"line":2584},[63,7781,92],{"emptyLinePlaceholder":91},[63,7783,7784,7786,7788,7790,7792,7794,7796],{"class":65,"line":2600},[63,7785,2722],{"class":196},[63,7787,2814],{"class":69},[63,7789,1078],{"class":2483},[63,7791,2731],{"class":2730},[63,7793,2490],{"class":2483},[63,7795,2487],{"class":2486},[63,7797,2496],{"class":2483},[63,7799,7800,7802,7804,7807,7809,7812,7814],{"class":65,"line":2608},[63,7801,2530],{"class":196},[63,7803,5287],{"class":81},[63,7805,7806],{"class":2483},".dense2(",[63,7808,2731],{"class":81},[63,7810,7811],{"class":2483},".relu(",[63,7813,2731],{"class":81},[63,7815,7816],{"class":2483},".dense1(X)))\n",[10,7818,7819],{"id":7819},"残差连接和层规范化",[14,7821,7822],{},"每个子层（自注意力或 FFN）外面都包了一层残差连接（residual connection）和层归一化（LayerNorm）。残差连接缓解了深层网络的梯度消失问题，让信息和梯度能够沿\"恒等通路\"穿透多层；而 LayerNorm 沿特征维度做归一化，相比 BatchNorm 不依赖 batch 大小，更适合变长序列的场景。",[14,7824,7825,7826,7951,7952,8071],{},"在原始论文（Post-LN）中，每个子层的输出形式为 ",[63,7827,7829,7897],{"className":7828},[342],[63,7830,7832],{"className":7831},[346],[348,7833,7834],{"xmlns":350},[352,7835,7836,7894],{},[355,7837,7838,7861,7863,7865,7867,7886,7888,7890,7892],{},[355,7839,7840,7843,7845,7848,7850,7853,7855,7857,7859],{},[371,7841,7842],{"mathvariant":1053},"L",[371,7844,38],{"mathvariant":1053},[371,7846,7847],{"mathvariant":1053},"y",[371,7849,1062],{"mathvariant":1053},[371,7851,7852],{"mathvariant":1053},"r",[371,7854,7342],{"mathvariant":1053},[371,7856,1072],{"mathvariant":1053},[371,7858,7852],{"mathvariant":1053},[371,7860,643],{"mathvariant":1053},[738,7862,1078],{"stretchy":1077},[371,7864,7347],{},[738,7866,5839],{},[355,7868,7869,7872,7874,7876,7878,7880,7882,7884],{},[371,7870,7871],{"mathvariant":1053},"S",[371,7873,2966],{"mathvariant":1053},[371,7875,7380],{"mathvariant":1053},[371,7877,2969],{"mathvariant":1053},[371,7879,38],{"mathvariant":1053},[371,7881,7847],{"mathvariant":1053},[371,7883,1062],{"mathvariant":1053},[371,7885,7852],{"mathvariant":1053},[738,7887,1078],{"stretchy":1077},[371,7889,7347],{},[738,7891,1092],{"stretchy":1077},[738,7893,1092],{"stretchy":1077},[378,7895,7896],{"encoding":380},"\\mathrm{LayerNorm}(x + \\mathrm{Sublayer}(x))",[63,7898,7900,7928],{"className":7899,"ariaHidden":386},[385],[63,7901,7903,7906,7913,7916,7919,7922,7925],{"className":7902},[390],[63,7904],{"className":7905,"style":1155},[394],[63,7907,7909],{"className":7908},[399],[63,7910,7912],{"className":7911},[399,1162],"LayerNorm",[63,7914,1078],{"className":7915},[403],[63,7917,7347],{"className":7918},[399,472],[63,7920],{"className":7921,"style":1920},[776],[63,7923,5839],{"className":7924},[830],[63,7926],{"className":7927,"style":1920},[776],[63,7929,7931,7934,7941,7944,7947],{"className":7930},[390],[63,7932],{"className":7933,"style":1155},[394],[63,7935,7937],{"className":7936},[399],[63,7938,7940],{"className":7939},[399,1162],"Sublayer",[63,7942,1078],{"className":7943},[403],[63,7945,7347],{"className":7946},[399,472],[63,7948,7950],{"className":7949},[593],"))","。我们在子层输出处先应用 dropout，再加回输入并归一化。注：现代实现中更常用 Pre-LN（即 ",[63,7953,7955,8019],{"className":7954},[342],[63,7956,7958],{"className":7957},[346],[348,7959,7960],{"xmlns":350},[352,7961,7962,8016],{},[355,7963,7964,7966,7968,7986,7988,8008,8010,8012,8014],{},[371,7965,7347],{},[738,7967,5839],{},[355,7969,7970,7972,7974,7976,7978,7980,7982,7984],{},[371,7971,7871],{"mathvariant":1053},[371,7973,2966],{"mathvariant":1053},[371,7975,7380],{"mathvariant":1053},[371,7977,2969],{"mathvariant":1053},[371,7979,38],{"mathvariant":1053},[371,7981,7847],{"mathvariant":1053},[371,7983,1062],{"mathvariant":1053},[371,7985,7852],{"mathvariant":1053},[738,7987,1078],{"stretchy":1077},[355,7989,7990,7992,7994,7996,7998,8000,8002,8004,8006],{},[371,7991,7842],{"mathvariant":1053},[371,7993,38],{"mathvariant":1053},[371,7995,7847],{"mathvariant":1053},[371,7997,1062],{"mathvariant":1053},[371,7999,7852],{"mathvariant":1053},[371,8001,7342],{"mathvariant":1053},[371,8003,1072],{"mathvariant":1053},[371,8005,7852],{"mathvariant":1053},[371,8007,643],{"mathvariant":1053},[738,8009,1078],{"stretchy":1077},[371,8011,7347],{},[738,8013,1092],{"stretchy":1077},[738,8015,1092],{"stretchy":1077},[378,8017,8018],{"encoding":380},"x + \\mathrm{Sublayer}(\\mathrm{LayerNorm}(x))",[63,8020,8022,8041],{"className":8021,"ariaHidden":386},[385],[63,8023,8025,8029,8032,8035,8038],{"className":8024},[390],[63,8026],{"className":8027,"style":8028},[394],"height:0.6667em;vertical-align:-0.0833em;",[63,8030,7347],{"className":8031},[399,472],[63,8033],{"className":8034,"style":1920},[776],[63,8036,5839],{"className":8037},[830],[63,8039],{"className":8040,"style":1920},[776],[63,8042,8044,8047,8053,8056,8062,8065,8068],{"className":8043},[390],[63,8045],{"className":8046,"style":1155},[394],[63,8048,8050],{"className":8049},[399],[63,8051,7940],{"className":8052},[399,1162],[63,8054,1078],{"className":8055},[403],[63,8057,8059],{"className":8058},[399],[63,8060,7912],{"className":8061},[399,1162],[63,8063,1078],{"className":8064},[403],[63,8066,7347],{"className":8067},[399,472],[63,8069,7950],{"className":8070},[593],"），训练更稳定，对 warmup 也不那么敏感。",[54,8073,8075],{"className":2468,"code":8074,"language":2470,"meta":59,"style":59},"class AddNorm(nn.Module):\n    \"\"\"残差连接后进行层规范化\"\"\"\n    def __init__(self, normalized_shape, dropout, **kwargs):\n        super(AddNorm, self).__init__(**kwargs)\n        self.dropout = nn.Dropout(dropout)\n        self.ln = nn.LayerNorm(normalized_shape)\n\n    def forward(self, X, Y):\n        return self.ln(self.dropout(Y) + X)\n",[49,8076,8077,8094,8099,8126,8145,8155,8167,8171,8192],{"__ignoreMap":59},[63,8078,8079,8081,8084,8086,8088,8090,8092],{"class":65,"line":66},[63,8080,2695],{"class":196},[63,8082,8083],{"class":2698}," AddNorm",[63,8085,1078],{"class":2483},[63,8087,2705],{"class":2704},[63,8089,1142],{"class":2483},[63,8091,2710],{"class":2704},[63,8093,2496],{"class":2483},[63,8095,8096],{"class":65,"line":88},[63,8097,8098],{"class":2501},"    \"\"\"残差连接后进行层规范化\"\"\"\n",[63,8100,8101,8103,8105,8107,8109,8111,8114,8116,8118,8120,8122,8124],{"class":65,"line":95},[63,8102,2722],{"class":196},[63,8104,2725],{"class":73},[63,8106,1078],{"class":2483},[63,8108,2731],{"class":2730},[63,8110,2490],{"class":2483},[63,8112,8113],{"class":2486},"normalized_shape",[63,8115,2490],{"class":2483},[63,8117,2736],{"class":2486},[63,8119,2490],{"class":2483},[63,8121,2741],{"class":196},[63,8123,2744],{"class":2486},[63,8125,2496],{"class":2483},[63,8127,8128,8130,8133,8135,8137,8139,8141,8143],{"class":65,"line":108},[63,8129,2752],{"class":2751},[63,8131,8132],{"class":2483},"(AddNorm, ",[63,8134,2731],{"class":81},[63,8136,2760],{"class":2483},[63,8138,2763],{"class":73},[63,8140,1078],{"class":2483},[63,8142,2741],{"class":196},[63,8144,2770],{"class":2483},[63,8146,8147,8149,8151,8153],{"class":65,"line":113},[63,8148,2775],{"class":81},[63,8150,2778],{"class":2483},[63,8152,204],{"class":196},[63,8154,2783],{"class":2483},[63,8156,8157,8159,8162,8164],{"class":65,"line":2548},[63,8158,2775],{"class":81},[63,8160,8161],{"class":2483},".ln ",[63,8163,204],{"class":196},[63,8165,8166],{"class":2483}," nn.LayerNorm(normalized_shape)\n",[63,8168,8169],{"class":65,"line":2556},[63,8170,92],{"emptyLinePlaceholder":91},[63,8172,8173,8175,8177,8179,8181,8183,8185,8187,8190],{"class":65,"line":2567},[63,8174,2722],{"class":196},[63,8176,2814],{"class":69},[63,8178,1078],{"class":2483},[63,8180,2731],{"class":2730},[63,8182,2490],{"class":2483},[63,8184,2487],{"class":2486},[63,8186,2490],{"class":2483},[63,8188,8189],{"class":2486},"Y",[63,8191,2496],{"class":2483},[63,8193,8194,8196,8198,8201,8203,8206,8208],{"class":65,"line":2584},[63,8195,2530],{"class":196},[63,8197,5287],{"class":81},[63,8199,8200],{"class":2483},".ln(",[63,8202,2731],{"class":81},[63,8204,8205],{"class":2483},".dropout(Y) ",[63,8207,5839],{"class":196},[63,8209,8210],{"class":2483}," X)\n",[10,8212,8213],{"id":8213},"编码器-解码器架构",[14,8215,8216,8217,8368,8369,8545,8546,8575,8576,8726],{},"主流的神经序列转换模型大多采用编码器-解码器结构。编码器把输入符号序列 ",[63,8218,8220,8254],{"className":8219},[342],[63,8221,8223],{"className":8222},[346],[348,8224,8225],{"xmlns":350},[352,8226,8227,8251],{},[355,8228,8229,8231,8237,8239,8241,8243,8249],{},[738,8230,1078],{"stretchy":1077},[368,8232,8233,8235],{},[371,8234,7347],{},[361,8236,363],{},[738,8238,1083],{"separator":386},[738,8240,3036],{},[738,8242,1083],{"separator":386},[368,8244,8245,8247],{},[371,8246,7347],{},[371,8248,612],{},[738,8250,1092],{"stretchy":1077},[378,8252,8253],{"encoding":380},"(x_1, \\ldots, x_n)",[63,8255,8257],{"className":8256,"ariaHidden":386},[385],[63,8258,8260,8263,8266,8306,8309,8312,8315,8318,8321,8324,8365],{"className":8259},[390],[63,8261],{"className":8262,"style":1155},[394],[63,8264,1078],{"className":8265},[403],[63,8267,8269,8272],{"className":8268},[399],[63,8270,7347],{"className":8271},[399,472],[63,8273,8275],{"className":8274},[476],[63,8276,8278,8298],{"className":8277},[411,412],[63,8279,8281,8295],{"className":8280},[416],[63,8282,8284],{"className":8283,"style":3157},[420],[63,8285,8286,8289],{"style":1535},[63,8287],{"className":8288,"style":817},[428],[63,8290,8292],{"className":8291},[433,434,435,436],[63,8293,363],{"className":8294},[399,436],[63,8296,507],{"className":8297},[506],[63,8299,8301],{"className":8300},[416],[63,8302,8304],{"className":8303,"style":1554},[420],[63,8305],{},[63,8307,1083],{"className":8308},[1173],[63,8310],{"className":8311,"style":1177},[776],[63,8313,3036],{"className":8314},[1222],[63,8316],{"className":8317,"style":1177},[776],[63,8319,1083],{"className":8320},[1173],[63,8322],{"className":8323,"style":1177},[776],[63,8325,8327,8330],{"className":8326},[399],[63,8328,7347],{"className":8329},[399,472],[63,8331,8333],{"className":8332},[476],[63,8334,8336,8357],{"className":8335},[411,412],[63,8337,8339,8354],{"className":8338},[416],[63,8340,8343],{"className":8341,"style":8342},[420],"height:0.1514em;",[63,8344,8345,8348],{"style":1535},[63,8346],{"className":8347,"style":817},[428],[63,8349,8351],{"className":8350},[433,434,435,436],[63,8352,612],{"className":8353},[399,472,436],[63,8355,507],{"className":8356},[506],[63,8358,8360],{"className":8359},[416],[63,8361,8363],{"className":8362,"style":1554},[420],[63,8364],{},[63,8366,1092],{"className":8367},[593]," 映射成连续表示 ",[63,8370,8372,8411],{"className":8371},[342],[63,8373,8375],{"className":8374},[346],[348,8376,8377],{"xmlns":350},[352,8378,8379,8408],{},[355,8380,8381,8384,8386,8388,8394,8396,8398,8400,8406],{},[371,8382,8383],{"mathvariant":5431},"z",[738,8385,204],{},[738,8387,1078],{"stretchy":1077},[368,8389,8390,8392],{},[371,8391,8383],{},[361,8393,363],{},[738,8395,1083],{"separator":386},[738,8397,3036],{},[738,8399,1083],{"separator":386},[368,8401,8402,8404],{},[371,8403,8383],{},[371,8405,612],{},[738,8407,1092],{"stretchy":1077},[378,8409,8410],{"encoding":380},"\\mathbf{z} = (z_1, \\ldots, z_n)",[63,8412,8414,8433],{"className":8413,"ariaHidden":386},[385],[63,8415,8417,8421,8424,8427,8430],{"className":8416},[390],[63,8418],{"className":8419,"style":8420},[394],"height:0.4444em;",[63,8422,8383],{"className":8423},[399,5466],[63,8425],{"className":8426,"style":777},[776],[63,8428,204],{"className":8429},[781],[63,8431],{"className":8432,"style":777},[776],[63,8434,8436,8439,8442,8484,8487,8490,8493,8496,8499,8502,8542],{"className":8435},[390],[63,8437],{"className":8438,"style":1155},[394],[63,8440,1078],{"className":8441},[403],[63,8443,8445,8449],{"className":8444},[399],[63,8446,8383],{"className":8447,"style":8448},[399,472],"margin-right:0.04398em;",[63,8450,8452],{"className":8451},[476],[63,8453,8455,8476],{"className":8454},[411,412],[63,8456,8458,8473],{"className":8457},[416],[63,8459,8461],{"className":8460,"style":3157},[420],[63,8462,8464,8467],{"style":8463},"top:-2.55em;margin-left:-0.044em;margin-right:0.05em;",[63,8465],{"className":8466,"style":817},[428],[63,8468,8470],{"className":8469},[433,434,435,436],[63,8471,363],{"className":8472},[399,436],[63,8474,507],{"className":8475},[506],[63,8477,8479],{"className":8478},[416],[63,8480,8482],{"className":8481,"style":1554},[420],[63,8483],{},[63,8485,1083],{"className":8486},[1173],[63,8488],{"className":8489,"style":1177},[776],[63,8491,3036],{"className":8492},[1222],[63,8494],{"className":8495,"style":1177},[776],[63,8497,1083],{"className":8498},[1173],[63,8500],{"className":8501,"style":1177},[776],[63,8503,8505,8508],{"className":8504},[399],[63,8506,8383],{"className":8507,"style":8448},[399,472],[63,8509,8511],{"className":8510},[476],[63,8512,8514,8534],{"className":8513},[411,412],[63,8515,8517,8531],{"className":8516},[416],[63,8518,8520],{"className":8519,"style":8342},[420],[63,8521,8522,8525],{"style":8463},[63,8523],{"className":8524,"style":817},[428],[63,8526,8528],{"className":8527},[433,434,435,436],[63,8529,612],{"className":8530},[399,472,436],[63,8532,507],{"className":8533},[506],[63,8535,8537],{"className":8536},[416],[63,8538,8540],{"className":8539,"style":1554},[420],[63,8541],{},[63,8543,1092],{"className":8544},[593],"；解码器再以 ",[63,8547,8549,8563],{"className":8548},[342],[63,8550,8552],{"className":8551},[346],[348,8553,8554],{"xmlns":350},[352,8555,8556,8560],{},[355,8557,8558],{},[371,8559,8383],{"mathvariant":5431},[378,8561,8562],{"encoding":380},"\\mathbf{z}",[63,8564,8566],{"className":8565,"ariaHidden":386},[385],[63,8567,8569,8572],{"className":8568},[390],[63,8570],{"className":8571,"style":8420},[394],[63,8573,8383],{"className":8574},[399,5466]," 为条件，自回归地一步步生成输出序列 ",[63,8577,8579,8613],{"className":8578},[342],[63,8580,8582],{"className":8581},[346],[348,8583,8584],{"xmlns":350},[352,8585,8586,8610],{},[355,8587,8588,8590,8596,8598,8600,8602,8608],{},[738,8589,1078],{"stretchy":1077},[368,8591,8592,8594],{},[371,8593,7847],{},[361,8595,363],{},[738,8597,1083],{"separator":386},[738,8599,3036],{},[738,8601,1083],{"separator":386},[368,8603,8604,8606],{},[371,8605,7847],{},[371,8607,643],{},[738,8609,1092],{"stretchy":1077},[378,8611,8612],{"encoding":380},"(y_1, \\ldots, y_m)",[63,8614,8616],{"className":8615,"ariaHidden":386},[385],[63,8617,8619,8622,8625,8665,8668,8671,8674,8677,8680,8683,8723],{"className":8618},[390],[63,8620],{"className":8621,"style":1155},[394],[63,8623,1078],{"className":8624},[403],[63,8626,8628,8631],{"className":8627},[399],[63,8629,7847],{"className":8630,"style":718},[399,472],[63,8632,8634],{"className":8633},[476],[63,8635,8637,8657],{"className":8636},[411,412],[63,8638,8640,8654],{"className":8639},[416],[63,8641,8643],{"className":8642,"style":3157},[420],[63,8644,8645,8648],{"style":2083},[63,8646],{"className":8647,"style":817},[428],[63,8649,8651],{"className":8650},[433,434,435,436],[63,8652,363],{"className":8653},[399,436],[63,8655,507],{"className":8656},[506],[63,8658,8660],{"className":8659},[416],[63,8661,8663],{"className":8662,"style":1554},[420],[63,8664],{},[63,8666,1083],{"className":8667},[1173],[63,8669],{"className":8670,"style":1177},[776],[63,8672,3036],{"className":8673},[1222],[63,8675],{"className":8676,"style":1177},[776],[63,8678,1083],{"className":8679},[1173],[63,8681],{"className":8682,"style":1177},[776],[63,8684,8686,8689],{"className":8685},[399],[63,8687,7847],{"className":8688,"style":718},[399,472],[63,8690,8692],{"className":8691},[476],[63,8693,8695,8715],{"className":8694},[411,412],[63,8696,8698,8712],{"className":8697},[416],[63,8699,8701],{"className":8700,"style":8342},[420],[63,8702,8703,8706],{"style":2083},[63,8704],{"className":8705,"style":817},[428],[63,8707,8709],{"className":8708},[433,434,435,436],[63,8710,643],{"className":8711},[399,472,436],[63,8713,507],{"className":8714},[506],[63,8716,8718],{"className":8717},[416],[63,8719,8721],{"className":8720,"style":1554},[420],[63,8722],{},[63,8724,1092],{"className":8725},[593],"——也就是说，每生成一个新词，都会把之前已经生成的所有词作为额外输入。",[54,8728,8730],{"className":2468,"code":8729,"language":2470,"meta":59,"style":59},"class Encoder(nn.Module):\n    \"\"\"编码器-解码器架构的基本编码器接口\"\"\"\n    def __init__(self, **kwargs):\n        super(Encoder, self).__init__(**kwargs)\n\n    def forward(self, X, *args):\n        raise NotImplementedError\n\nclass Decoder(nn.Module):\n    \"\"\"编码器-解码器架构的基本解码器接口\"\"\"\n    def __init__(self, **kwargs):\n        super(Decoder, self).__init__(**kwargs)\n\n    def init_state(self, enc_outputs, *args):\n        raise NotImplementedError\n\n    def forward(self, X, state):\n        raise NotImplementedError\n\nclass EncoderDecoder(nn.Module):\n    \"\"\"编码器-解码器架构的基类\"\"\"\n    def __init__(self, encoder, decoder, **kwargs):\n        super(EncoderDecoder, self).__init__(**kwargs)\n        self.encoder = encoder\n        self.decoder = decoder\n\n    def forward(self, enc_X, dec_X, *args):\n        enc_outputs = self.encoder(enc_X, *args)\n        dec_state = self.decoder.init_state(enc_outputs, *args)\n        return self.decoder(dec_X, dec_state)\n",[49,8731,8732,8749,8754,8772,8791,8795,8818,8826,8830,8847,8852,8870,8889,8893,8917,8923,8927,8948,8954,8958,8975,8980,9008,9027,9039,9051,9055,9083,9100,9116],{"__ignoreMap":59},[63,8733,8734,8736,8739,8741,8743,8745,8747],{"class":65,"line":66},[63,8735,2695],{"class":196},[63,8737,8738],{"class":2698}," Encoder",[63,8740,1078],{"class":2483},[63,8742,2705],{"class":2704},[63,8744,1142],{"class":2483},[63,8746,2710],{"class":2704},[63,8748,2496],{"class":2483},[63,8750,8751],{"class":65,"line":88},[63,8752,8753],{"class":2501},"    \"\"\"编码器-解码器架构的基本编码器接口\"\"\"\n",[63,8755,8756,8758,8760,8762,8764,8766,8768,8770],{"class":65,"line":95},[63,8757,2722],{"class":196},[63,8759,2725],{"class":73},[63,8761,1078],{"class":2483},[63,8763,2731],{"class":2730},[63,8765,2490],{"class":2483},[63,8767,2741],{"class":196},[63,8769,2744],{"class":2486},[63,8771,2496],{"class":2483},[63,8773,8774,8776,8779,8781,8783,8785,8787,8789],{"class":65,"line":108},[63,8775,2752],{"class":2751},[63,8777,8778],{"class":2483},"(Encoder, ",[63,8780,2731],{"class":81},[63,8782,2760],{"class":2483},[63,8784,2763],{"class":73},[63,8786,1078],{"class":2483},[63,8788,2741],{"class":196},[63,8790,2770],{"class":2483},[63,8792,8793],{"class":65,"line":113},[63,8794,92],{"emptyLinePlaceholder":91},[63,8796,8797,8799,8801,8803,8805,8807,8809,8811,8813,8816],{"class":65,"line":2548},[63,8798,2722],{"class":196},[63,8800,2814],{"class":69},[63,8802,1078],{"class":2483},[63,8804,2731],{"class":2730},[63,8806,2490],{"class":2483},[63,8808,2487],{"class":2486},[63,8810,2490],{"class":2483},[63,8812,82],{"class":196},[63,8814,8815],{"class":2486},"args",[63,8817,2496],{"class":2483},[63,8819,8820,8823],{"class":65,"line":2556},[63,8821,8822],{"class":196},"        raise",[63,8824,8825],{"class":2751}," NotImplementedError\n",[63,8827,8828],{"class":65,"line":2567},[63,8829,92],{"emptyLinePlaceholder":91},[63,8831,8832,8834,8837,8839,8841,8843,8845],{"class":65,"line":2584},[63,8833,2695],{"class":196},[63,8835,8836],{"class":2698}," Decoder",[63,8838,1078],{"class":2483},[63,8840,2705],{"class":2704},[63,8842,1142],{"class":2483},[63,8844,2710],{"class":2704},[63,8846,2496],{"class":2483},[63,8848,8849],{"class":65,"line":2600},[63,8850,8851],{"class":2501},"    \"\"\"编码器-解码器架构的基本解码器接口\"\"\"\n",[63,8853,8854,8856,8858,8860,8862,8864,8866,8868],{"class":65,"line":2608},[63,8855,2722],{"class":196},[63,8857,2725],{"class":73},[63,8859,1078],{"class":2483},[63,8861,2731],{"class":2730},[63,8863,2490],{"class":2483},[63,8865,2741],{"class":196},[63,8867,2744],{"class":2486},[63,8869,2496],{"class":2483},[63,8871,8872,8874,8877,8879,8881,8883,8885,8887],{"class":65,"line":2625},[63,8873,2752],{"class":2751},[63,8875,8876],{"class":2483},"(Decoder, ",[63,8878,2731],{"class":81},[63,8880,2760],{"class":2483},[63,8882,2763],{"class":73},[63,8884,1078],{"class":2483},[63,8886,2741],{"class":196},[63,8888,2770],{"class":2483},[63,8890,8891],{"class":65,"line":2631},[63,8892,92],{"emptyLinePlaceholder":91},[63,8894,8895,8897,8900,8902,8904,8906,8909,8911,8913,8915],{"class":65,"line":2656},[63,8896,2722],{"class":196},[63,8898,8899],{"class":69}," init_state",[63,8901,1078],{"class":2483},[63,8903,2731],{"class":2730},[63,8905,2490],{"class":2483},[63,8907,8908],{"class":2486},"enc_outputs",[63,8910,2490],{"class":2483},[63,8912,82],{"class":196},[63,8914,8815],{"class":2486},[63,8916,2496],{"class":2483},[63,8918,8919,8921],{"class":65,"line":2669},[63,8920,8822],{"class":196},[63,8922,8825],{"class":2751},[63,8924,8925],{"class":65,"line":2907},[63,8926,92],{"emptyLinePlaceholder":91},[63,8928,8929,8931,8933,8935,8937,8939,8941,8943,8946],{"class":65,"line":4848},[63,8930,2722],{"class":196},[63,8932,2814],{"class":69},[63,8934,1078],{"class":2483},[63,8936,2731],{"class":2730},[63,8938,2490],{"class":2483},[63,8940,2487],{"class":2486},[63,8942,2490],{"class":2483},[63,8944,8945],{"class":2486},"state",[63,8947,2496],{"class":2483},[63,8949,8950,8952],{"class":65,"line":5124},[63,8951,8822],{"class":196},[63,8953,8825],{"class":2751},[63,8955,8956],{"class":65,"line":5130},[63,8957,92],{"emptyLinePlaceholder":91},[63,8959,8960,8962,8965,8967,8969,8971,8973],{"class":65,"line":5136},[63,8961,2695],{"class":196},[63,8963,8964],{"class":2698}," EncoderDecoder",[63,8966,1078],{"class":2483},[63,8968,2705],{"class":2704},[63,8970,1142],{"class":2483},[63,8972,2710],{"class":2704},[63,8974,2496],{"class":2483},[63,8976,8977],{"class":65,"line":5142},[63,8978,8979],{"class":2501},"    \"\"\"编码器-解码器架构的基类\"\"\"\n",[63,8981,8982,8984,8986,8988,8990,8992,8995,8997,9000,9002,9004,9006],{"class":65,"line":5163},[63,8983,2722],{"class":196},[63,8985,2725],{"class":73},[63,8987,1078],{"class":2483},[63,8989,2731],{"class":2730},[63,8991,2490],{"class":2483},[63,8993,8994],{"class":2486},"encoder",[63,8996,2490],{"class":2483},[63,8998,8999],{"class":2486},"decoder",[63,9001,2490],{"class":2483},[63,9003,2741],{"class":196},[63,9005,2744],{"class":2486},[63,9007,2496],{"class":2483},[63,9009,9010,9012,9015,9017,9019,9021,9023,9025],{"class":65,"line":5182},[63,9011,2752],{"class":2751},[63,9013,9014],{"class":2483},"(EncoderDecoder, ",[63,9016,2731],{"class":81},[63,9018,2760],{"class":2483},[63,9020,2763],{"class":73},[63,9022,1078],{"class":2483},[63,9024,2741],{"class":196},[63,9026,2770],{"class":2483},[63,9028,9029,9031,9034,9036],{"class":65,"line":5201},[63,9030,2775],{"class":81},[63,9032,9033],{"class":2483},".encoder ",[63,9035,204],{"class":196},[63,9037,9038],{"class":2483}," encoder\n",[63,9040,9041,9043,9046,9048],{"class":65,"line":5206},[63,9042,2775],{"class":81},[63,9044,9045],{"class":2483},".decoder ",[63,9047,204],{"class":196},[63,9049,9050],{"class":2483}," decoder\n",[63,9052,9053],{"class":65,"line":5222},[63,9054,92],{"emptyLinePlaceholder":91},[63,9056,9057,9059,9061,9063,9065,9067,9070,9072,9075,9077,9079,9081],{"class":65,"line":5228},[63,9058,2722],{"class":196},[63,9060,2814],{"class":69},[63,9062,1078],{"class":2483},[63,9064,2731],{"class":2730},[63,9066,2490],{"class":2483},[63,9068,9069],{"class":2486},"enc_X",[63,9071,2490],{"class":2483},[63,9073,9074],{"class":2486},"dec_X",[63,9076,2490],{"class":2483},[63,9078,82],{"class":196},[63,9080,8815],{"class":2486},[63,9082,2496],{"class":2483},[63,9084,9085,9088,9090,9092,9095,9097],{"class":65,"line":5234},[63,9086,9087],{"class":2483},"        enc_outputs ",[63,9089,204],{"class":196},[63,9091,5287],{"class":81},[63,9093,9094],{"class":2483},".encoder(enc_X, ",[63,9096,82],{"class":196},[63,9098,9099],{"class":2483},"args)\n",[63,9101,9102,9105,9107,9109,9112,9114],{"class":65,"line":5244},[63,9103,9104],{"class":2483},"        dec_state ",[63,9106,204],{"class":196},[63,9108,5287],{"class":81},[63,9110,9111],{"class":2483},".decoder.init_state(enc_outputs, ",[63,9113,82],{"class":196},[63,9115,9099],{"class":2483},[63,9117,9118,9120,9122],{"class":65,"line":5268},[63,9119,2530],{"class":196},[63,9121,5287],{"class":81},[63,9123,9124],{"class":2483},".decoder(dec_X, dec_state)\n",[10,9126,9128],{"id":9127},"transformer-编码器","transformer 编码器",[14,9130,9131],{},"代码实现如下：",[54,9133,9135],{"className":2468,"code":9134,"language":2470,"meta":59,"style":59},"class EncoderBlock(nn.Module):\n    \"\"\"Transformer编码器块\"\"\"\n    def __init__(self, key_size, query_size, value_size, num_hiddens,\n                 norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,\n                 dropout, use_bias=False, **kwargs):\n        super(EncoderBlock, self).__init__(**kwargs)\n        self.attention = d2l.MultiHeadAttention(\n            key_size, query_size, value_size, num_hiddens, num_heads, dropout,\n            use_bias)\n        self.addnorm1 = AddNorm(norm_shape, dropout)\n        self.ffn = PositionWiseFFN(\n            ffn_num_input, ffn_num_hiddens, num_hiddens)\n        self.addnorm2 = AddNorm(norm_shape, dropout)\n\n    def forward(self, X, valid_lens):\n        Y = self.addnorm1(X, self.attention(X, X, X, valid_lens))\n        return self.addnorm2(Y, self.ffn(Y))\n\nclass TransformerEncoder(Encoder):\n    \"\"\"Transformer编码器\"\"\"\n    def __init__(self, vocab_size, key_size, query_size, value_size,\n                 num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens,\n                 num_heads, num_layers, dropout, use_bias=False, **kwargs):\n        super(TransformerEncoder, self).__init__(**kwargs)\n        self.num_hiddens = num_hiddens\n        self.embedding = nn.Embedding(vocab_size, num_hiddens)\n        self.pos_encoding = PositionalEncoding(num_hiddens, dropout)\n        self.blks = nn.Sequential()\n        for i in range(num_layers):\n            self.blks.add_module(\"block\"+str(i),\n                EncoderBlock(key_size, query_size, value_size, num_hiddens,\n                             norm_shape, ffn_num_input, ffn_num_hiddens,\n                             num_heads, dropout, use_bias))\n\n    def forward(self, X, valid_lens, *args):\n        # 因为位置编码值在-1和1之间，\n        # 因此嵌入值乘以嵌入维度的平方根进行缩放，\n        # 然后再与位置编码相加。\n        X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\n        self.attention_weights = [None] * len(self.blks)\n        for i, blk in enumerate(self.blks):\n            X = blk(X, valid_lens)\n            self.attention_weights[i] = blk.attention.attention.attention_weights\n        return X\n",[49,9136,9137,9154,9159,9187,9206,9228,9247,9258,9263,9268,9280,9292,9297,9308,9312,9332,9349,9363,9367,9381,9386,9415,9435,9464,9483,9495,9507,9519,9531,9549,9572,9577,9582,9587,9591,9617,9622,9628,9634,9661,9689,9709,9720,9733],{"__ignoreMap":59},[63,9138,9139,9141,9144,9146,9148,9150,9152],{"class":65,"line":66},[63,9140,2695],{"class":196},[63,9142,9143],{"class":2698}," EncoderBlock",[63,9145,1078],{"class":2483},[63,9147,2705],{"class":2704},[63,9149,1142],{"class":2483},[63,9151,2710],{"class":2704},[63,9153,2496],{"class":2483},[63,9155,9156],{"class":65,"line":88},[63,9157,9158],{"class":2501},"    \"\"\"Transformer编码器块\"\"\"\n",[63,9160,9161,9163,9165,9167,9169,9171,9173,9175,9177,9179,9181,9183,9185],{"class":65,"line":95},[63,9162,2722],{"class":196},[63,9164,2725],{"class":73},[63,9166,1078],{"class":2483},[63,9168,2731],{"class":2730},[63,9170,2490],{"class":2483},[63,9172,4914],{"class":2486},[63,9174,2490],{"class":2483},[63,9176,4919],{"class":2486},[63,9178,2490],{"class":2483},[63,9180,4924],{"class":2486},[63,9182,2490],{"class":2483},[63,9184,4929],{"class":2486},[63,9186,3891],{"class":2483},[63,9188,9189,9192,9194,9196,9198,9200,9202,9204],{"class":65,"line":108},[63,9190,9191],{"class":2486},"                 norm_shape",[63,9193,2490],{"class":2483},[63,9195,7701],{"class":2486},[63,9197,2490],{"class":2483},[63,9199,7706],{"class":2486},[63,9201,2490],{"class":2483},[63,9203,4662],{"class":2486},[63,9205,3891],{"class":2483},[63,9207,9208,9211,9213,9216,9218,9220,9222,9224,9226],{"class":65,"line":113},[63,9209,9210],{"class":2486},"                 dropout",[63,9212,2490],{"class":2483},[63,9214,9215],{"class":2486},"use_bias",[63,9217,204],{"class":196},[63,9219,4950],{"class":73},[63,9221,2490],{"class":2483},[63,9223,2741],{"class":196},[63,9225,2744],{"class":2486},[63,9227,2496],{"class":2483},[63,9229,9230,9232,9235,9237,9239,9241,9243,9245],{"class":65,"line":2548},[63,9231,2752],{"class":2751},[63,9233,9234],{"class":2483},"(EncoderBlock, ",[63,9236,2731],{"class":81},[63,9238,2760],{"class":2483},[63,9240,2763],{"class":73},[63,9242,1078],{"class":2483},[63,9244,2741],{"class":196},[63,9246,2770],{"class":2483},[63,9248,9249,9251,9253,9255],{"class":65,"line":2556},[63,9250,2775],{"class":81},[63,9252,4996],{"class":2483},[63,9254,204],{"class":196},[63,9256,9257],{"class":2483}," d2l.MultiHeadAttention(\n",[63,9259,9260],{"class":65,"line":2567},[63,9261,9262],{"class":2483},"            key_size, query_size, value_size, num_hiddens, num_heads, dropout,\n",[63,9264,9265],{"class":65,"line":2584},[63,9266,9267],{"class":2483},"            use_bias)\n",[63,9269,9270,9272,9275,9277],{"class":65,"line":2600},[63,9271,2775],{"class":81},[63,9273,9274],{"class":2483},".addnorm1 ",[63,9276,204],{"class":196},[63,9278,9279],{"class":2483}," AddNorm(norm_shape, dropout)\n",[63,9281,9282,9284,9287,9289],{"class":65,"line":2608},[63,9283,2775],{"class":81},[63,9285,9286],{"class":2483},".ffn ",[63,9288,204],{"class":196},[63,9290,9291],{"class":2483}," PositionWiseFFN(\n",[63,9293,9294],{"class":65,"line":2625},[63,9295,9296],{"class":2483},"            ffn_num_input, ffn_num_hiddens, num_hiddens)\n",[63,9298,9299,9301,9304,9306],{"class":65,"line":2631},[63,9300,2775],{"class":81},[63,9302,9303],{"class":2483},".addnorm2 ",[63,9305,204],{"class":196},[63,9307,9279],{"class":2483},[63,9309,9310],{"class":65,"line":2656},[63,9311,92],{"emptyLinePlaceholder":91},[63,9313,9314,9316,9318,9320,9322,9324,9326,9328,9330],{"class":65,"line":2669},[63,9315,2722],{"class":196},[63,9317,2814],{"class":69},[63,9319,1078],{"class":2483},[63,9321,2731],{"class":2730},[63,9323,2490],{"class":2483},[63,9325,2487],{"class":2486},[63,9327,2490],{"class":2483},[63,9329,2493],{"class":2486},[63,9331,2496],{"class":2483},[63,9333,9334,9337,9339,9341,9344,9346],{"class":65,"line":2907},[63,9335,9336],{"class":2483},"        Y ",[63,9338,204],{"class":196},[63,9340,5287],{"class":81},[63,9342,9343],{"class":2483},".addnorm1(X, ",[63,9345,2731],{"class":81},[63,9347,9348],{"class":2483},".attention(X, X, X, valid_lens))\n",[63,9350,9351,9353,9355,9358,9360],{"class":65,"line":4848},[63,9352,2530],{"class":196},[63,9354,5287],{"class":81},[63,9356,9357],{"class":2483},".addnorm2(Y, ",[63,9359,2731],{"class":81},[63,9361,9362],{"class":2483},".ffn(Y))\n",[63,9364,9365],{"class":65,"line":5124},[63,9366,92],{"emptyLinePlaceholder":91},[63,9368,9369,9371,9374,9376,9379],{"class":65,"line":5130},[63,9370,2695],{"class":196},[63,9372,9373],{"class":2698}," TransformerEncoder",[63,9375,1078],{"class":2483},[63,9377,9378],{"class":2704},"Encoder",[63,9380,2496],{"class":2483},[63,9382,9383],{"class":65,"line":5136},[63,9384,9385],{"class":2501},"    \"\"\"Transformer编码器\"\"\"\n",[63,9387,9388,9390,9392,9394,9396,9398,9401,9403,9405,9407,9409,9411,9413],{"class":65,"line":5142},[63,9389,2722],{"class":196},[63,9391,2725],{"class":73},[63,9393,1078],{"class":2483},[63,9395,2731],{"class":2730},[63,9397,2490],{"class":2483},[63,9399,9400],{"class":2486},"vocab_size",[63,9402,2490],{"class":2483},[63,9404,4914],{"class":2486},[63,9406,2490],{"class":2483},[63,9408,4919],{"class":2486},[63,9410,2490],{"class":2483},[63,9412,4924],{"class":2486},[63,9414,3891],{"class":2483},[63,9416,9417,9420,9422,9425,9427,9429,9431,9433],{"class":65,"line":5163},[63,9418,9419],{"class":2486},"                 num_hiddens",[63,9421,2490],{"class":2483},[63,9423,9424],{"class":2486},"norm_shape",[63,9426,2490],{"class":2483},[63,9428,7701],{"class":2486},[63,9430,2490],{"class":2483},[63,9432,7706],{"class":2486},[63,9434,3891],{"class":2483},[63,9436,9437,9439,9441,9444,9446,9448,9450,9452,9454,9456,9458,9460,9462],{"class":65,"line":5182},[63,9438,4936],{"class":2486},[63,9440,2490],{"class":2483},[63,9442,9443],{"class":2486},"num_layers",[63,9445,2490],{"class":2483},[63,9447,2736],{"class":2486},[63,9449,2490],{"class":2483},[63,9451,9215],{"class":2486},[63,9453,204],{"class":196},[63,9455,4950],{"class":73},[63,9457,2490],{"class":2483},[63,9459,2741],{"class":196},[63,9461,2744],{"class":2486},[63,9463,2496],{"class":2483},[63,9465,9466,9468,9471,9473,9475,9477,9479,9481],{"class":65,"line":5201},[63,9467,2752],{"class":2751},[63,9469,9470],{"class":2483},"(TransformerEncoder, ",[63,9472,2731],{"class":81},[63,9474,2760],{"class":2483},[63,9476,2763],{"class":73},[63,9478,1078],{"class":2483},[63,9480,2741],{"class":196},[63,9482,2770],{"class":2483},[63,9484,9485,9487,9490,9492],{"class":65,"line":5206},[63,9486,2775],{"class":81},[63,9488,9489],{"class":2483},".num_hiddens ",[63,9491,204],{"class":196},[63,9493,9494],{"class":2483}," num_hiddens\n",[63,9496,9497,9499,9502,9504],{"class":65,"line":5222},[63,9498,2775],{"class":81},[63,9500,9501],{"class":2483},".embedding ",[63,9503,204],{"class":196},[63,9505,9506],{"class":2483}," nn.Embedding(vocab_size, num_hiddens)\n",[63,9508,9509,9511,9514,9516],{"class":65,"line":5228},[63,9510,2775],{"class":81},[63,9512,9513],{"class":2483},".pos_encoding ",[63,9515,204],{"class":196},[63,9517,9518],{"class":2483}," PositionalEncoding(num_hiddens, dropout)\n",[63,9520,9521,9523,9526,9528],{"class":65,"line":5234},[63,9522,2775],{"class":81},[63,9524,9525],{"class":2483},".blks ",[63,9527,204],{"class":196},[63,9529,9530],{"class":2483}," nn.Sequential()\n",[63,9532,9533,9536,9539,9542,9546],{"class":65,"line":5244},[63,9534,9535],{"class":196},"        for",[63,9537,9538],{"class":2483}," i ",[63,9540,9541],{"class":196},"in",[63,9543,9545],{"class":9544},"sDgm9"," range",[63,9547,9548],{"class":2483},"(num_layers):\n",[63,9550,9551,9554,9557,9559,9562,9564,9566,9569],{"class":65,"line":5268},[63,9552,9553],{"class":81},"            self",[63,9555,9556],{"class":2483},".blks.add_module(",[63,9558,208],{"class":207},[63,9560,9561],{"class":77},"block",[63,9563,208],{"class":207},[63,9565,5839],{"class":196},[63,9567,9568],{"class":2751},"str",[63,9570,9571],{"class":2483},"(i),\n",[63,9573,9574],{"class":65,"line":5273},[63,9575,9576],{"class":2483},"                EncoderBlock(key_size, query_size, value_size, num_hiddens,\n",[63,9578,9579],{"class":65,"line":5279},[63,9580,9581],{"class":2483},"                             norm_shape, ffn_num_input, ffn_num_hiddens,\n",[63,9583,9584],{"class":65,"line":5293},[63,9585,9586],{"class":2483},"                             num_heads, dropout, use_bias))\n",[63,9588,9589],{"class":65,"line":5298},[63,9590,92],{"emptyLinePlaceholder":91},[63,9592,9593,9595,9597,9599,9601,9603,9605,9607,9609,9611,9613,9615],{"class":65,"line":5304},[63,9594,2722],{"class":196},[63,9596,2814],{"class":69},[63,9598,1078],{"class":2483},[63,9600,2731],{"class":2730},[63,9602,2490],{"class":2483},[63,9604,2487],{"class":2486},[63,9606,2490],{"class":2483},[63,9608,2493],{"class":2486},[63,9610,2490],{"class":2483},[63,9612,82],{"class":196},[63,9614,8815],{"class":2486},[63,9616,2496],{"class":2483},[63,9618,9619],{"class":65,"line":5319},[63,9620,9621],{"class":2507},"        # 因为位置编码值在-1和1之间，\n",[63,9623,9625],{"class":65,"line":9624},37,[63,9626,9627],{"class":2507},"        # 因此嵌入值乘以嵌入维度的平方根进行缩放，\n",[63,9629,9631],{"class":65,"line":9630},38,[63,9632,9633],{"class":2507},"        # 然后再与位置编码相加。\n",[63,9635,9637,9639,9641,9643,9646,9648,9651,9653,9656,9658],{"class":65,"line":9636},39,[63,9638,2634],{"class":2483},[63,9640,204],{"class":196},[63,9642,5287],{"class":81},[63,9644,9645],{"class":2483},".pos_encoding(",[63,9647,2731],{"class":81},[63,9649,9650],{"class":2483},".embedding(X) ",[63,9652,82],{"class":196},[63,9654,9655],{"class":2483}," math.sqrt(",[63,9657,2731],{"class":81},[63,9659,9660],{"class":2483},".num_hiddens))\n",[63,9662,9664,9666,9668,9670,9673,9675,9677,9679,9682,9684,9686],{"class":65,"line":9663},40,[63,9665,2775],{"class":81},[63,9667,2899],{"class":2483},[63,9669,204],{"class":196},[63,9671,9672],{"class":2483}," [",[63,9674,2842],{"class":73},[63,9676,7218],{"class":2483},[63,9678,82],{"class":196},[63,9680,9681],{"class":9544}," len",[63,9683,1078],{"class":2483},[63,9685,2731],{"class":81},[63,9687,9688],{"class":2483},".blks)\n",[63,9690,9692,9694,9697,9699,9702,9704,9706],{"class":65,"line":9691},41,[63,9693,9535],{"class":196},[63,9695,9696],{"class":2483}," i, blk ",[63,9698,9541],{"class":196},[63,9700,9701],{"class":9544}," enumerate",[63,9703,1078],{"class":2483},[63,9705,2731],{"class":81},[63,9707,9708],{"class":2483},".blks):\n",[63,9710,9712,9715,9717],{"class":65,"line":9711},42,[63,9713,9714],{"class":2483},"            X ",[63,9716,204],{"class":196},[63,9718,9719],{"class":2483}," blk(X, valid_lens)\n",[63,9721,9723,9725,9728,9730],{"class":65,"line":9722},43,[63,9724,9553],{"class":81},[63,9726,9727],{"class":2483},".attention_weights[i] ",[63,9729,204],{"class":196},[63,9731,9732],{"class":2483}," blk.attention.attention.attention_weights\n",[63,9734,9736,9738],{"class":65,"line":9735},44,[63,9737,2530],{"class":196},[63,9739,9740],{"class":2483}," X\n",[10,9742,9744],{"id":9743},"transformer-解码器","transformer 解码器",[14,9746,9747],{},"代码如下：",[54,9749,9751],{"className":2468,"code":9750,"language":2470,"meta":59,"style":59},"class DecoderBlock(nn.Module):\n    \"\"\"解码器中第i个块\"\"\"\n    def __init__(self, key_size, query_size, value_size, num_hiddens,\n                 norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,\n                 dropout, i, **kwargs):\n        super(DecoderBlock, self).__init__(**kwargs)\n        self.i = i\n        self.attention1 = d2l.MultiHeadAttention(\n            key_size, query_size, value_size, num_hiddens, num_heads, dropout)\n        self.addnorm1 = AddNorm(norm_shape, dropout)\n        self.attention2 = d2l.MultiHeadAttention(\n            key_size, query_size, value_size, num_hiddens, num_heads, dropout)\n        self.addnorm2 = AddNorm(norm_shape, dropout)\n        self.ffn = PositionWiseFFN(ffn_num_input, ffn_num_hiddens,\n                                   num_hiddens)\n        self.addnorm3 = AddNorm(norm_shape, dropout)\n\n    def forward(self, X, state):\n        enc_outputs, enc_valid_lens = state[0], state[1]\n        # 训练阶段，输出序列的所有词元都在同一时间处理，\n        # 因此state[2][self.i]初始化为None。\n        # 预测阶段，输出序列是通过词元一个接着一个解码的，\n        # 因此state[2][self.i]包含着直到当前时间步第i个块解码的输出表示\n        if state[2][self.i] is None:\n            key_values = X\n        else:\n            key_values = torch.cat((state[2][self.i], X), axis=1)\n        state[2][self.i] = key_values\n        if self.training:\n            batch_size, num_steps, _ = X.shape\n            # dec_valid_lens的开头:(batch_size,num_steps),\n            # 其中每一行是[1,2,...,num_steps]\n            dec_valid_lens = torch.arange(\n                1, num_steps + 1, device=X.device).repeat(batch_size, 1)\n        else:\n            dec_valid_lens = None\n\n        # 自注意力\n        X2 = self.attention1(X, key_values, key_values, dec_valid_lens)\n        Y = self.addnorm1(X, X2)\n        # 编码器－解码器注意力。\n        # enc_outputs的开头:(batch_size,num_steps,num_hiddens)\n        Y2 = self.attention2(Y, enc_outputs, enc_outputs, enc_valid_lens)\n        Z = self.addnorm2(Y, Y2)\n        return self.addnorm3(Z, self.ffn(Z)), state\n\nclass TransformerDecoder(AttentionDecoder):\n    def __init__(self, vocab_size, key_size, query_size, value_size,\n                 num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens,\n                 num_heads, num_layers, dropout, **kwargs):\n        super(TransformerDecoder, self).__init__(**kwargs)\n        self.num_hiddens = num_hiddens\n        self.num_layers = num_layers\n        self.embedding = nn.Embedding(vocab_size, num_hiddens)\n        self.pos_encoding = PositionalEncoding(num_hiddens, dropout)\n        self.blks = nn.Sequential()\n        for i in range(num_layers):\n            self.blks.add_module(\"block\"+str(i),\n                DecoderBlock(key_size, query_size, value_size, num_hiddens,\n                             norm_shape, ffn_num_input, ffn_num_hiddens,\n                             num_heads, dropout, i))\n        self.dense = nn.Linear(num_hiddens, vocab_size)\n\n    def init_state(self, enc_outputs, enc_valid_lens, *args):\n        return [enc_outputs, enc_valid_lens, [None] * self.num_layers]\n\n    def forward(self, X, state):\n        X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\n        self._attention_weights = [[None] * len(self.blks) for _ in range (2)]\n        for i, blk in enumerate(self.blks):\n            X, state = blk(X, state)\n            # 解码器自注意力权重\n            self._attention_weights[0][i] = blk.attention1.attention.attention_weights\n            # “编码器－解码器”自注意力权重\n            self._attention_weights[1][i] = blk.attention2.attention.attention_weights\n        return self.dense(X), state\n\n    @property\n    def attention_weights(self):\n        return self._attention_weights\n",[49,9752,9753,9770,9775,9803,9821,9837,9856,9868,9879,9884,9894,9905,9909,9919,9930,9935,9946,9950,9970,9989,9994,9999,10004,10009,10031,10040,10046,10073,10091,10100,10109,10114,10119,10129,10155,10161,10170,10174,10179,10191,10202,10207,10212,10224,10236,10251,10256,10271,10300,10319,10340,10360,10371,10384,10395,10406,10417,10430,10449,10455,10460,10466,10479,10484,10512,10531,10536,10557,10580,10626,10643,10654,10660,10678,10684,10700,10710,10715,10724,10738],{"__ignoreMap":59},[63,9754,9755,9757,9760,9762,9764,9766,9768],{"class":65,"line":66},[63,9756,2695],{"class":196},[63,9758,9759],{"class":2698}," DecoderBlock",[63,9761,1078],{"class":2483},[63,9763,2705],{"class":2704},[63,9765,1142],{"class":2483},[63,9767,2710],{"class":2704},[63,9769,2496],{"class":2483},[63,9771,9772],{"class":65,"line":88},[63,9773,9774],{"class":2501},"    \"\"\"解码器中第i个块\"\"\"\n",[63,9776,9777,9779,9781,9783,9785,9787,9789,9791,9793,9795,9797,9799,9801],{"class":65,"line":95},[63,9778,2722],{"class":196},[63,9780,2725],{"class":73},[63,9782,1078],{"class":2483},[63,9784,2731],{"class":2730},[63,9786,2490],{"class":2483},[63,9788,4914],{"class":2486},[63,9790,2490],{"class":2483},[63,9792,4919],{"class":2486},[63,9794,2490],{"class":2483},[63,9796,4924],{"class":2486},[63,9798,2490],{"class":2483},[63,9800,4929],{"class":2486},[63,9802,3891],{"class":2483},[63,9804,9805,9807,9809,9811,9813,9815,9817,9819],{"class":65,"line":108},[63,9806,9191],{"class":2486},[63,9808,2490],{"class":2483},[63,9810,7701],{"class":2486},[63,9812,2490],{"class":2483},[63,9814,7706],{"class":2486},[63,9816,2490],{"class":2483},[63,9818,4662],{"class":2486},[63,9820,3891],{"class":2483},[63,9822,9823,9825,9827,9829,9831,9833,9835],{"class":65,"line":113},[63,9824,9210],{"class":2486},[63,9826,2490],{"class":2483},[63,9828,1069],{"class":2486},[63,9830,2490],{"class":2483},[63,9832,2741],{"class":196},[63,9834,2744],{"class":2486},[63,9836,2496],{"class":2483},[63,9838,9839,9841,9844,9846,9848,9850,9852,9854],{"class":65,"line":2548},[63,9840,2752],{"class":2751},[63,9842,9843],{"class":2483},"(DecoderBlock, ",[63,9845,2731],{"class":81},[63,9847,2760],{"class":2483},[63,9849,2763],{"class":73},[63,9851,1078],{"class":2483},[63,9853,2741],{"class":196},[63,9855,2770],{"class":2483},[63,9857,9858,9860,9863,9865],{"class":65,"line":2556},[63,9859,2775],{"class":81},[63,9861,9862],{"class":2483},".i ",[63,9864,204],{"class":196},[63,9866,9867],{"class":2483}," i\n",[63,9869,9870,9872,9875,9877],{"class":65,"line":2567},[63,9871,2775],{"class":81},[63,9873,9874],{"class":2483},".attention1 ",[63,9876,204],{"class":196},[63,9878,9257],{"class":2483},[63,9880,9881],{"class":65,"line":2584},[63,9882,9883],{"class":2483},"            key_size, query_size, value_size, num_hiddens, num_heads, dropout)\n",[63,9885,9886,9888,9890,9892],{"class":65,"line":2600},[63,9887,2775],{"class":81},[63,9889,9274],{"class":2483},[63,9891,204],{"class":196},[63,9893,9279],{"class":2483},[63,9895,9896,9898,9901,9903],{"class":65,"line":2608},[63,9897,2775],{"class":81},[63,9899,9900],{"class":2483},".attention2 ",[63,9902,204],{"class":196},[63,9904,9257],{"class":2483},[63,9906,9907],{"class":65,"line":2625},[63,9908,9883],{"class":2483},[63,9910,9911,9913,9915,9917],{"class":65,"line":2631},[63,9912,2775],{"class":81},[63,9914,9303],{"class":2483},[63,9916,204],{"class":196},[63,9918,9279],{"class":2483},[63,9920,9921,9923,9925,9927],{"class":65,"line":2656},[63,9922,2775],{"class":81},[63,9924,9286],{"class":2483},[63,9926,204],{"class":196},[63,9928,9929],{"class":2483}," PositionWiseFFN(ffn_num_input, ffn_num_hiddens,\n",[63,9931,9932],{"class":65,"line":2669},[63,9933,9934],{"class":2483},"                                   num_hiddens)\n",[63,9936,9937,9939,9942,9944],{"class":65,"line":2907},[63,9938,2775],{"class":81},[63,9940,9941],{"class":2483},".addnorm3 ",[63,9943,204],{"class":196},[63,9945,9279],{"class":2483},[63,9947,9948],{"class":65,"line":4848},[63,9949,92],{"emptyLinePlaceholder":91},[63,9951,9952,9954,9956,9958,9960,9962,9964,9966,9968],{"class":65,"line":5124},[63,9953,2722],{"class":196},[63,9955,2814],{"class":69},[63,9957,1078],{"class":2483},[63,9959,2731],{"class":2730},[63,9961,2490],{"class":2483},[63,9963,2487],{"class":2486},[63,9965,2490],{"class":2483},[63,9967,8945],{"class":2486},[63,9969,2496],{"class":2483},[63,9971,9972,9975,9977,9980,9982,9985,9987],{"class":65,"line":5130},[63,9973,9974],{"class":2483},"        enc_outputs, enc_valid_lens ",[63,9976,204],{"class":196},[63,9978,9979],{"class":2483}," state[",[63,9981,1802],{"class":73},[63,9983,9984],{"class":2483},"], state[",[63,9986,363],{"class":73},[63,9988,2861],{"class":2483},[63,9990,9991],{"class":65,"line":5136},[63,9992,9993],{"class":2507},"        # 训练阶段，输出序列的所有词元都在同一时间处理，\n",[63,9995,9996],{"class":65,"line":5142},[63,9997,9998],{"class":2507},"        # 因此state[2][self.i]初始化为None。\n",[63,10000,10001],{"class":65,"line":5163},[63,10002,10003],{"class":2507},"        # 预测阶段，输出序列是通过词元一个接着一个解码的，\n",[63,10005,10006],{"class":65,"line":5182},[63,10007,10008],{"class":2507},"        # 因此state[2][self.i]包含着直到当前时间步第i个块解码的输出表示\n",[63,10010,10011,10013,10015,10017,10020,10022,10025,10027,10029],{"class":65,"line":5201},[63,10012,2570],{"class":196},[63,10014,9979],{"class":2483},[63,10016,2883],{"class":73},[63,10018,10019],{"class":2483},"][",[63,10021,2731],{"class":81},[63,10023,10024],{"class":2483},".i] ",[63,10026,2519],{"class":196},[63,10028,2522],{"class":73},[63,10030,2525],{"class":2483},[63,10032,10033,10036,10038],{"class":65,"line":5206},[63,10034,10035],{"class":2483},"            key_values ",[63,10037,204],{"class":196},[63,10039,9740],{"class":2483},[63,10041,10042,10044],{"class":65,"line":5222},[63,10043,2603],{"class":196},[63,10045,2525],{"class":2483},[63,10047,10048,10050,10052,10055,10057,10059,10061,10064,10067,10069,10071],{"class":65,"line":5228},[63,10049,10035],{"class":2483},[63,10051,204],{"class":196},[63,10053,10054],{"class":2483}," torch.cat((state[",[63,10056,2883],{"class":73},[63,10058,10019],{"class":2483},[63,10060,2731],{"class":81},[63,10062,10063],{"class":2483},".i], X), ",[63,10065,10066],{"class":2536},"axis",[63,10068,204],{"class":196},[63,10070,363],{"class":73},[63,10072,2545],{"class":2483},[63,10074,10075,10078,10080,10082,10084,10086,10088],{"class":65,"line":5234},[63,10076,10077],{"class":2483},"        state[",[63,10079,2883],{"class":73},[63,10081,10019],{"class":2483},[63,10083,2731],{"class":81},[63,10085,10024],{"class":2483},[63,10087,204],{"class":196},[63,10089,10090],{"class":2483}," key_values\n",[63,10092,10093,10095,10097],{"class":65,"line":5244},[63,10094,2570],{"class":196},[63,10096,5287],{"class":81},[63,10098,10099],{"class":2483},".training:\n",[63,10101,10102,10105,10107],{"class":65,"line":5268},[63,10103,10104],{"class":2483},"            batch_size, num_steps, _ ",[63,10106,204],{"class":196},[63,10108,2564],{"class":2483},[63,10110,10111],{"class":65,"line":5273},[63,10112,10113],{"class":2507},"            # dec_valid_lens的开头:(batch_size,num_steps),\n",[63,10115,10116],{"class":65,"line":5279},[63,10117,10118],{"class":2507},"            # 其中每一行是[1,2,...,num_steps]\n",[63,10120,10121,10124,10126],{"class":65,"line":5293},[63,10122,10123],{"class":2483},"            dec_valid_lens ",[63,10125,204],{"class":196},[63,10127,10128],{"class":2483}," torch.arange(\n",[63,10130,10131,10134,10137,10139,10141,10143,10146,10148,10151,10153],{"class":65,"line":5298},[63,10132,10133],{"class":73},"                1",[63,10135,10136],{"class":2483},", num_steps ",[63,10138,5839],{"class":196},[63,10140,2579],{"class":73},[63,10142,2490],{"class":2483},[63,10144,10145],{"class":2536},"device",[63,10147,204],{"class":196},[63,10149,10150],{"class":2483},"X.device).repeat(batch_size, ",[63,10152,363],{"class":73},[63,10154,2545],{"class":2483},[63,10156,10157,10159],{"class":65,"line":5304},[63,10158,2603],{"class":196},[63,10160,2525],{"class":2483},[63,10162,10163,10165,10167],{"class":65,"line":5319},[63,10164,10123],{"class":2483},[63,10166,204],{"class":196},[63,10168,10169],{"class":73}," None\n",[63,10171,10172],{"class":65,"line":9624},[63,10173,92],{"emptyLinePlaceholder":91},[63,10175,10176],{"class":65,"line":9630},[63,10177,10178],{"class":2507},"        # 自注意力\n",[63,10180,10181,10184,10186,10188],{"class":65,"line":9636},[63,10182,10183],{"class":2483},"        X2 ",[63,10185,204],{"class":196},[63,10187,5287],{"class":81},[63,10189,10190],{"class":2483},".attention1(X, key_values, key_values, dec_valid_lens)\n",[63,10192,10193,10195,10197,10199],{"class":65,"line":9663},[63,10194,9336],{"class":2483},[63,10196,204],{"class":196},[63,10198,5287],{"class":81},[63,10200,10201],{"class":2483},".addnorm1(X, X2)\n",[63,10203,10204],{"class":65,"line":9691},[63,10205,10206],{"class":2507},"        # 编码器－解码器注意力。\n",[63,10208,10209],{"class":65,"line":9711},[63,10210,10211],{"class":2507},"        # enc_outputs的开头:(batch_size,num_steps,num_hiddens)\n",[63,10213,10214,10217,10219,10221],{"class":65,"line":9722},[63,10215,10216],{"class":2483},"        Y2 ",[63,10218,204],{"class":196},[63,10220,5287],{"class":81},[63,10222,10223],{"class":2483},".attention2(Y, enc_outputs, enc_outputs, enc_valid_lens)\n",[63,10225,10226,10229,10231,10233],{"class":65,"line":9735},[63,10227,10228],{"class":2483},"        Z ",[63,10230,204],{"class":196},[63,10232,5287],{"class":81},[63,10234,10235],{"class":2483},".addnorm2(Y, Y2)\n",[63,10237,10239,10241,10243,10246,10248],{"class":65,"line":10238},45,[63,10240,2530],{"class":196},[63,10242,5287],{"class":81},[63,10244,10245],{"class":2483},".addnorm3(Z, ",[63,10247,2731],{"class":81},[63,10249,10250],{"class":2483},".ffn(Z)), state\n",[63,10252,10254],{"class":65,"line":10253},46,[63,10255,92],{"emptyLinePlaceholder":91},[63,10257,10259,10261,10264,10266,10269],{"class":65,"line":10258},47,[63,10260,2695],{"class":196},[63,10262,10263],{"class":2698}," TransformerDecoder",[63,10265,1078],{"class":2483},[63,10267,10268],{"class":2704},"AttentionDecoder",[63,10270,2496],{"class":2483},[63,10272,10274,10276,10278,10280,10282,10284,10286,10288,10290,10292,10294,10296,10298],{"class":65,"line":10273},48,[63,10275,2722],{"class":196},[63,10277,2725],{"class":73},[63,10279,1078],{"class":2483},[63,10281,2731],{"class":2730},[63,10283,2490],{"class":2483},[63,10285,9400],{"class":2486},[63,10287,2490],{"class":2483},[63,10289,4914],{"class":2486},[63,10291,2490],{"class":2483},[63,10293,4919],{"class":2486},[63,10295,2490],{"class":2483},[63,10297,4924],{"class":2486},[63,10299,3891],{"class":2483},[63,10301,10303,10305,10307,10309,10311,10313,10315,10317],{"class":65,"line":10302},49,[63,10304,9419],{"class":2486},[63,10306,2490],{"class":2483},[63,10308,9424],{"class":2486},[63,10310,2490],{"class":2483},[63,10312,7701],{"class":2486},[63,10314,2490],{"class":2483},[63,10316,7706],{"class":2486},[63,10318,3891],{"class":2483},[63,10320,10322,10324,10326,10328,10330,10332,10334,10336,10338],{"class":65,"line":10321},50,[63,10323,4936],{"class":2486},[63,10325,2490],{"class":2483},[63,10327,9443],{"class":2486},[63,10329,2490],{"class":2483},[63,10331,2736],{"class":2486},[63,10333,2490],{"class":2483},[63,10335,2741],{"class":196},[63,10337,2744],{"class":2486},[63,10339,2496],{"class":2483},[63,10341,10343,10345,10348,10350,10352,10354,10356,10358],{"class":65,"line":10342},51,[63,10344,2752],{"class":2751},[63,10346,10347],{"class":2483},"(TransformerDecoder, ",[63,10349,2731],{"class":81},[63,10351,2760],{"class":2483},[63,10353,2763],{"class":73},[63,10355,1078],{"class":2483},[63,10357,2741],{"class":196},[63,10359,2770],{"class":2483},[63,10361,10363,10365,10367,10369],{"class":65,"line":10362},52,[63,10364,2775],{"class":81},[63,10366,9489],{"class":2483},[63,10368,204],{"class":196},[63,10370,9494],{"class":2483},[63,10372,10374,10376,10379,10381],{"class":65,"line":10373},53,[63,10375,2775],{"class":81},[63,10377,10378],{"class":2483},".num_layers ",[63,10380,204],{"class":196},[63,10382,10383],{"class":2483}," num_layers\n",[63,10385,10387,10389,10391,10393],{"class":65,"line":10386},54,[63,10388,2775],{"class":81},[63,10390,9501],{"class":2483},[63,10392,204],{"class":196},[63,10394,9506],{"class":2483},[63,10396,10398,10400,10402,10404],{"class":65,"line":10397},55,[63,10399,2775],{"class":81},[63,10401,9513],{"class":2483},[63,10403,204],{"class":196},[63,10405,9518],{"class":2483},[63,10407,10409,10411,10413,10415],{"class":65,"line":10408},56,[63,10410,2775],{"class":81},[63,10412,9525],{"class":2483},[63,10414,204],{"class":196},[63,10416,9530],{"class":2483},[63,10418,10420,10422,10424,10426,10428],{"class":65,"line":10419},57,[63,10421,9535],{"class":196},[63,10423,9538],{"class":2483},[63,10425,9541],{"class":196},[63,10427,9545],{"class":9544},[63,10429,9548],{"class":2483},[63,10431,10433,10435,10437,10439,10441,10443,10445,10447],{"class":65,"line":10432},58,[63,10434,9553],{"class":81},[63,10436,9556],{"class":2483},[63,10438,208],{"class":207},[63,10440,9561],{"class":77},[63,10442,208],{"class":207},[63,10444,5839],{"class":196},[63,10446,9568],{"class":2751},[63,10448,9571],{"class":2483},[63,10450,10452],{"class":65,"line":10451},59,[63,10453,10454],{"class":2483},"                DecoderBlock(key_size, query_size, value_size, num_hiddens,\n",[63,10456,10458],{"class":65,"line":10457},60,[63,10459,9581],{"class":2483},[63,10461,10463],{"class":65,"line":10462},61,[63,10464,10465],{"class":2483},"                             num_heads, dropout, i))\n",[63,10467,10469,10471,10474,10476],{"class":65,"line":10468},62,[63,10470,2775],{"class":81},[63,10472,10473],{"class":2483},".dense ",[63,10475,204],{"class":196},[63,10477,10478],{"class":2483}," nn.Linear(num_hiddens, vocab_size)\n",[63,10480,10482],{"class":65,"line":10481},63,[63,10483,92],{"emptyLinePlaceholder":91},[63,10485,10487,10489,10491,10493,10495,10497,10499,10501,10504,10506,10508,10510],{"class":65,"line":10486},64,[63,10488,2722],{"class":196},[63,10490,8899],{"class":69},[63,10492,1078],{"class":2483},[63,10494,2731],{"class":2730},[63,10496,2490],{"class":2483},[63,10498,8908],{"class":2486},[63,10500,2490],{"class":2483},[63,10502,10503],{"class":2486},"enc_valid_lens",[63,10505,2490],{"class":2483},[63,10507,82],{"class":196},[63,10509,8815],{"class":2486},[63,10511,2496],{"class":2483},[63,10513,10515,10517,10520,10522,10524,10526,10528],{"class":65,"line":10514},65,[63,10516,2530],{"class":196},[63,10518,10519],{"class":2483}," [enc_outputs, enc_valid_lens, [",[63,10521,2842],{"class":73},[63,10523,7218],{"class":2483},[63,10525,82],{"class":196},[63,10527,5287],{"class":81},[63,10529,10530],{"class":2483},".num_layers]\n",[63,10532,10534],{"class":65,"line":10533},66,[63,10535,92],{"emptyLinePlaceholder":91},[63,10537,10539,10541,10543,10545,10547,10549,10551,10553,10555],{"class":65,"line":10538},67,[63,10540,2722],{"class":196},[63,10542,2814],{"class":69},[63,10544,1078],{"class":2483},[63,10546,2731],{"class":2730},[63,10548,2490],{"class":2483},[63,10550,2487],{"class":2486},[63,10552,2490],{"class":2483},[63,10554,8945],{"class":2486},[63,10556,2496],{"class":2483},[63,10558,10560,10562,10564,10566,10568,10570,10572,10574,10576,10578],{"class":65,"line":10559},68,[63,10561,2634],{"class":2483},[63,10563,204],{"class":196},[63,10565,5287],{"class":81},[63,10567,9645],{"class":2483},[63,10569,2731],{"class":81},[63,10571,9650],{"class":2483},[63,10573,82],{"class":196},[63,10575,9655],{"class":2483},[63,10577,2731],{"class":81},[63,10579,9660],{"class":2483},[63,10581,10583,10585,10588,10590,10593,10595,10597,10599,10601,10603,10605,10608,10611,10614,10616,10618,10621,10623],{"class":65,"line":10582},69,[63,10584,2775],{"class":81},[63,10586,10587],{"class":2483},"._attention_weights ",[63,10589,204],{"class":196},[63,10591,10592],{"class":2483}," [[",[63,10594,2842],{"class":73},[63,10596,7218],{"class":2483},[63,10598,82],{"class":196},[63,10600,9681],{"class":9544},[63,10602,1078],{"class":2483},[63,10604,2731],{"class":81},[63,10606,10607],{"class":2483},".blks) ",[63,10609,10610],{"class":196},"for",[63,10612,10613],{"class":2483}," _ ",[63,10615,9541],{"class":196},[63,10617,9545],{"class":9544},[63,10619,10620],{"class":2483}," (",[63,10622,2883],{"class":73},[63,10624,10625],{"class":2483},")]\n",[63,10627,10629,10631,10633,10635,10637,10639,10641],{"class":65,"line":10628},70,[63,10630,9535],{"class":196},[63,10632,9696],{"class":2483},[63,10634,9541],{"class":196},[63,10636,9701],{"class":9544},[63,10638,1078],{"class":2483},[63,10640,2731],{"class":81},[63,10642,9708],{"class":2483},[63,10644,10646,10649,10651],{"class":65,"line":10645},71,[63,10647,10648],{"class":2483},"            X, state ",[63,10650,204],{"class":196},[63,10652,10653],{"class":2483}," blk(X, state)\n",[63,10655,10657],{"class":65,"line":10656},72,[63,10658,10659],{"class":2507},"            # 解码器自注意力权重\n",[63,10661,10663,10665,10668,10670,10673,10675],{"class":65,"line":10662},73,[63,10664,9553],{"class":81},[63,10666,10667],{"class":2483},"._attention_weights[",[63,10669,1802],{"class":73},[63,10671,10672],{"class":2483},"][i] ",[63,10674,204],{"class":196},[63,10676,10677],{"class":2483}," blk.attention1.attention.attention_weights\n",[63,10679,10681],{"class":65,"line":10680},74,[63,10682,10683],{"class":2507},"            # “编码器－解码器”自注意力权重\n",[63,10685,10687,10689,10691,10693,10695,10697],{"class":65,"line":10686},75,[63,10688,9553],{"class":81},[63,10690,10667],{"class":2483},[63,10692,363],{"class":73},[63,10694,10672],{"class":2483},[63,10696,204],{"class":196},[63,10698,10699],{"class":2483}," blk.attention2.attention.attention_weights\n",[63,10701,10703,10705,10707],{"class":65,"line":10702},76,[63,10704,2530],{"class":196},[63,10706,5287],{"class":81},[63,10708,10709],{"class":2483},".dense(X), state\n",[63,10711,10713],{"class":65,"line":10712},77,[63,10714,92],{"emptyLinePlaceholder":91},[63,10716,10718,10721],{"class":65,"line":10717},78,[63,10719,10720],{"class":69},"    @",[63,10722,10723],{"class":2751},"property\n",[63,10725,10727,10729,10732,10734,10736],{"class":65,"line":10726},79,[63,10728,2722],{"class":196},[63,10730,10731],{"class":69}," attention_weights",[63,10733,1078],{"class":2483},[63,10735,2731],{"class":2730},[63,10737,2496],{"class":2483},[63,10739,10741,10743,10745],{"class":65,"line":10740},80,[63,10742,2530],{"class":196},[63,10744,5287],{"class":81},[63,10746,10747],{"class":2483},"._attention_weights\n",[10,10749,10750],{"id":10750},"训练",[14,10752,10753],{},"训练时同样需要屏蔽 padding 等无效位置，避免它们污染损失值：",[54,10755,10757],{"className":2468,"code":10756,"language":2470,"meta":59,"style":59},"def sequence_mask(X, valid_len, value=0):\n    \"\"\"在序列中屏蔽不相关的项\"\"\"\n    maxlen = X.size(1)\n    mask = torch.arange((maxlen), dtype=torch.float32,\n                        device=X.device)[None, :] \u003C valid_len[:, None]\n    X[~mask] = value\n    return X\n\nclass MaskedSoftmaxCELoss(nn.CrossEntropyLoss):\n    \"\"\"带遮蔽的softmax交叉熵损失函数\"\"\"\n    # pred的形状：(batch_size,num_steps,vocab_size)\n    # label的形状：(batch_size,num_steps)\n    # valid_len的形状：(batch_size,)\n    def forward(self, pred, label, valid_len):\n        weights = torch.ones_like(label)\n        weights = sequence_mask(weights, valid_len)\n        self.reduction='none'\n        unweighted_loss = super(MaskedSoftmaxCELoss, self).forward(\n            pred.permute(0, 2, 1), label)\n        weighted_loss = (unweighted_loss * weights).mean(dim=1)\n        return weighted_loss\n",[49,10758,10759,10786,10791,10805,10823,10852,10868,10874,10878,10896,10901,10906,10911,10916,10942,10952,10961,10979,10997,11015,11038],{"__ignoreMap":59},[63,10760,10761,10763,10766,10768,10770,10772,10775,10777,10780,10782,10784],{"class":65,"line":66},[63,10762,2477],{"class":196},[63,10764,10765],{"class":69}," sequence_mask",[63,10767,1078],{"class":2483},[63,10769,2487],{"class":2486},[63,10771,2490],{"class":2483},[63,10773,10774],{"class":2486},"valid_len",[63,10776,2490],{"class":2483},[63,10778,10779],{"class":2486},"value",[63,10781,204],{"class":196},[63,10783,1802],{"class":73},[63,10785,2496],{"class":2483},[63,10787,10788],{"class":65,"line":88},[63,10789,10790],{"class":2501},"    \"\"\"在序列中屏蔽不相关的项\"\"\"\n",[63,10792,10793,10796,10798,10801,10803],{"class":65,"line":95},[63,10794,10795],{"class":2483},"    maxlen ",[63,10797,204],{"class":196},[63,10799,10800],{"class":2483}," X.size(",[63,10802,363],{"class":73},[63,10804,2545],{"class":2483},[63,10806,10807,10810,10812,10815,10818,10820],{"class":65,"line":108},[63,10808,10809],{"class":2483},"    mask ",[63,10811,204],{"class":196},[63,10813,10814],{"class":2483}," torch.arange((maxlen), ",[63,10816,10817],{"class":2536},"dtype",[63,10819,204],{"class":196},[63,10821,10822],{"class":2483},"torch.float32,\n",[63,10824,10825,10828,10830,10833,10835,10838,10841,10844,10846,10848,10850],{"class":65,"line":113},[63,10826,10827],{"class":2536},"                        device",[63,10829,204],{"class":196},[63,10831,10832],{"class":2483},"X.device)[",[63,10834,2842],{"class":73},[63,10836,10837],{"class":2483},", :] ",[63,10839,10840],{"class":196},"\u003C",[63,10842,10843],{"class":2483}," valid_len[",[63,10845,7202],{"class":7201},[63,10847,2490],{"class":2483},[63,10849,2842],{"class":73},[63,10851,2861],{"class":2483},[63,10853,10854,10857,10860,10863,10865],{"class":65,"line":2548},[63,10855,10856],{"class":2483},"    X[",[63,10858,10859],{"class":196},"~",[63,10861,10862],{"class":2483},"mask] ",[63,10864,204],{"class":196},[63,10866,10867],{"class":2483}," value\n",[63,10869,10870,10872],{"class":65,"line":2556},[63,10871,4754],{"class":196},[63,10873,9740],{"class":2483},[63,10875,10876],{"class":65,"line":2567},[63,10877,92],{"emptyLinePlaceholder":91},[63,10879,10880,10882,10885,10887,10889,10891,10894],{"class":65,"line":2584},[63,10881,2695],{"class":196},[63,10883,10884],{"class":2698}," MaskedSoftmaxCELoss",[63,10886,1078],{"class":2483},[63,10888,2705],{"class":2704},[63,10890,1142],{"class":2483},[63,10892,10893],{"class":2704},"CrossEntropyLoss",[63,10895,2496],{"class":2483},[63,10897,10898],{"class":65,"line":2600},[63,10899,10900],{"class":2501},"    \"\"\"带遮蔽的softmax交叉熵损失函数\"\"\"\n",[63,10902,10903],{"class":65,"line":2608},[63,10904,10905],{"class":2507},"    # pred的形状：(batch_size,num_steps,vocab_size)\n",[63,10907,10908],{"class":65,"line":2625},[63,10909,10910],{"class":2507},"    # label的形状：(batch_size,num_steps)\n",[63,10912,10913],{"class":65,"line":2631},[63,10914,10915],{"class":2507},"    # valid_len的形状：(batch_size,)\n",[63,10917,10918,10920,10922,10924,10926,10928,10931,10933,10936,10938,10940],{"class":65,"line":2656},[63,10919,2722],{"class":196},[63,10921,2814],{"class":69},[63,10923,1078],{"class":2483},[63,10925,2731],{"class":2730},[63,10927,2490],{"class":2483},[63,10929,10930],{"class":2486},"pred",[63,10932,2490],{"class":2483},[63,10934,10935],{"class":2486},"label",[63,10937,2490],{"class":2483},[63,10939,10774],{"class":2486},[63,10941,2496],{"class":2483},[63,10943,10944,10947,10949],{"class":65,"line":2669},[63,10945,10946],{"class":2483},"        weights ",[63,10948,204],{"class":196},[63,10950,10951],{"class":2483}," torch.ones_like(label)\n",[63,10953,10954,10956,10958],{"class":65,"line":2907},[63,10955,10946],{"class":2483},[63,10957,204],{"class":196},[63,10959,10960],{"class":2483}," sequence_mask(weights, valid_len)\n",[63,10962,10963,10965,10968,10970,10973,10976],{"class":65,"line":4848},[63,10964,2775],{"class":81},[63,10966,10967],{"class":2483},".reduction",[63,10969,204],{"class":196},[63,10971,10972],{"class":207},"'",[63,10974,10975],{"class":77},"none",[63,10977,10978],{"class":207},"'\n",[63,10980,10981,10984,10986,10989,10992,10994],{"class":65,"line":5124},[63,10982,10983],{"class":2483},"        unweighted_loss ",[63,10985,204],{"class":196},[63,10987,10988],{"class":2751}," super",[63,10990,10991],{"class":2483},"(MaskedSoftmaxCELoss, ",[63,10993,2731],{"class":81},[63,10995,10996],{"class":2483},").forward(\n",[63,10998,10999,11002,11004,11006,11008,11010,11012],{"class":65,"line":5130},[63,11000,11001],{"class":2483},"            pred.permute(",[63,11003,1802],{"class":73},[63,11005,2490],{"class":2483},[63,11007,2883],{"class":73},[63,11009,2490],{"class":2483},[63,11011,363],{"class":73},[63,11013,11014],{"class":2483},"), label)\n",[63,11016,11017,11020,11022,11025,11027,11030,11032,11034,11036],{"class":65,"line":5136},[63,11018,11019],{"class":2483},"        weighted_loss ",[63,11021,204],{"class":196},[63,11023,11024],{"class":2483}," (unweighted_loss ",[63,11026,82],{"class":196},[63,11028,11029],{"class":2483}," weights).mean(",[63,11031,2537],{"class":2536},[63,11033,204],{"class":196},[63,11035,363],{"class":73},[63,11037,2545],{"class":2483},[63,11039,11040,11042],{"class":65,"line":5142},[63,11041,2530],{"class":196},[63,11043,11044],{"class":2483}," weighted_loss\n",[14,11046,11047],{},"梯度裁剪函数：",[54,11049,11051],{"className":2468,"code":11050,"language":2470,"meta":59,"style":59},"def grad_clipping(net, theta):\n    \"\"\"裁剪梯度\"\"\"\n    if isinstance(net, nn.Module):\n        params = [p for p in net.parameters() if p.requires_grad]\n    else:\n        params = net.params\n    norm = torch.sqrt(sum(torch.sum((p.grad ** 2)) for p in params))\n    if norm > theta:\n        for param in params:\n            param.grad[:] *= theta / norm\n",[49,11052,11053,11072,11077,11087,11113,11119,11128,11160,11173,11185],{"__ignoreMap":59},[63,11054,11055,11057,11060,11062,11065,11067,11070],{"class":65,"line":66},[63,11056,2477],{"class":196},[63,11058,11059],{"class":69}," grad_clipping",[63,11061,1078],{"class":2483},[63,11063,11064],{"class":2486},"net",[63,11066,2490],{"class":2483},[63,11068,11069],{"class":2486},"theta",[63,11071,2496],{"class":2483},[63,11073,11074],{"class":65,"line":88},[63,11075,11076],{"class":2501},"    \"\"\"裁剪梯度\"\"\"\n",[63,11078,11079,11081,11084],{"class":65,"line":95},[63,11080,2513],{"class":196},[63,11082,11083],{"class":9544}," isinstance",[63,11085,11086],{"class":2483},"(net, nn.Module):\n",[63,11088,11089,11092,11094,11097,11099,11102,11104,11107,11110],{"class":65,"line":108},[63,11090,11091],{"class":2483},"        params ",[63,11093,204],{"class":196},[63,11095,11096],{"class":2483}," [p ",[63,11098,10610],{"class":196},[63,11100,11101],{"class":2483}," p ",[63,11103,9541],{"class":196},[63,11105,11106],{"class":2483}," net.parameters() ",[63,11108,11109],{"class":196},"if",[63,11111,11112],{"class":2483}," p.requires_grad]\n",[63,11114,11115,11117],{"class":65,"line":113},[63,11116,2551],{"class":196},[63,11118,2525],{"class":2483},[63,11120,11121,11123,11125],{"class":65,"line":2548},[63,11122,11091],{"class":2483},[63,11124,204],{"class":196},[63,11126,11127],{"class":2483}," net.params\n",[63,11129,11130,11133,11135,11138,11141,11144,11146,11149,11151,11153,11155,11157],{"class":65,"line":2556},[63,11131,11132],{"class":2483},"    norm ",[63,11134,204],{"class":196},[63,11136,11137],{"class":2483}," torch.sqrt(",[63,11139,11140],{"class":9544},"sum",[63,11142,11143],{"class":2483},"(torch.sum((p.grad ",[63,11145,2741],{"class":196},[63,11147,11148],{"class":73}," 2",[63,11150,2886],{"class":2483},[63,11152,10610],{"class":196},[63,11154,11101],{"class":2483},[63,11156,9541],{"class":196},[63,11158,11159],{"class":2483}," params))\n",[63,11161,11162,11164,11167,11170],{"class":65,"line":2567},[63,11163,2513],{"class":196},[63,11165,11166],{"class":2483}," norm ",[63,11168,11169],{"class":196},">",[63,11171,11172],{"class":2483}," theta:\n",[63,11174,11175,11177,11180,11182],{"class":65,"line":2584},[63,11176,9535],{"class":196},[63,11178,11179],{"class":2483}," param ",[63,11181,9541],{"class":196},[63,11183,11184],{"class":2483}," params:\n",[63,11186,11187,11190,11192,11194,11197,11200,11202],{"class":65,"line":2600},[63,11188,11189],{"class":2483},"            param.grad[",[63,11191,7202],{"class":7201},[63,11193,7218],{"class":2483},[63,11195,11196],{"class":196},"*=",[63,11198,11199],{"class":2483}," theta ",[63,11201,2889],{"class":196},[63,11203,11204],{"class":2483}," norm\n",[14,11206,11207,11208,11211,11212,11215,11216,11220],{},"在训练时，会把特殊的开始符 ",[49,11209,11210],{},"\u003Cbos>"," 与原始输出序列（去掉结束符 ",[49,11213,11214],{},"\u003Ceos>","）拼接，作为解码器的输入——这种做法称为 ",[11217,11218,11219],"strong",{},"强制教学（teacher forcing）","：直接将真实标签喂给解码器，避免训练早期因预测漂移导致的误差累积；与之相对的另一种选择，是让上一时间步预测出的 token 作为下一时间步的输入。",[54,11222,11224],{"className":2468,"code":11223,"language":2470,"meta":59,"style":59},"def train_seq2seq(net, data_iter, lr, num_epochs, tgt_vocab, device):\n    \"\"\"训练序列到序列模型\"\"\"\n    def xavier_init_weights(m):\n        if type(m) == nn.Linear:\n            nn.init.xavier_uniform_(m.weight)\n        if type(m) == nn.GRU:\n            for param in m._flat_weights_names:\n                if \"weight\" in param:\n                    nn.init.xavier_uniform_(m._parameters[param])\n\n    net.apply(xavier_init_weights)\n    net.to(device)\n    optimizer = torch.optim.Adam(net.parameters(), lr=lr)\n    loss = MaskedSoftmaxCELoss()\n    net.train()\n    animator = d2l.Animator(xlabel='epoch', ylabel='loss',\n                     xlim=[10, num_epochs])\n    for epoch in range(num_epochs):\n        timer = d2l.Timer()\n        metric = d2l.Accumulator(2)  # 训练损失总和，词元数量\n        for batch in data_iter:\n            optimizer.zero_grad()\n            X, X_valid_len, Y, Y_valid_len = [x.to(device) for x in batch]\n            bos = torch.tensor([tgt_vocab['\u003Cbos>']] * Y.shape[0],\n                          device=device).reshape(-1, 1)\n            dec_input = torch.cat([bos, Y[:, :-1]], 1)  # 强制教学\n            Y_hat, _ = net(X, dec_input, X_valid_len)\n            l = loss(Y_hat, Y, Y_valid_len)\n            l.sum().backward()      # 损失函数的标量进行“反向传播”\n            grad_clipping(net, 1)\n            num_tokens = Y_valid_len.sum()\n            optimizer.step()\n            with torch.no_grad():\n                metric.add(l.sum(), num_tokens)\n        if (epoch + 1) % 10 == 0:\n            animator.add(epoch + 1, (metric[0] / metric[1],))\n    print(f'loss {metric[0] / metric[1]:.3f}, {metric[1] / timer.stop():.1f} '\n        f'tokens/sec on {str(device)}')\n",[49,11225,11226,11263,11268,11281,11296,11301,11319,11331,11350,11355,11359,11364,11369,11386,11396,11401,11439,11455,11470,11480,11498,11510,11515,11535,11564,11584,11614,11624,11634,11642,11651,11661,11666,11674,11679,11706,11732,11793],{"__ignoreMap":59},[63,11227,11228,11230,11233,11235,11237,11239,11242,11244,11247,11249,11252,11254,11257,11259,11261],{"class":65,"line":66},[63,11229,2477],{"class":196},[63,11231,11232],{"class":69}," train_seq2seq",[63,11234,1078],{"class":2483},[63,11236,11064],{"class":2486},[63,11238,2490],{"class":2483},[63,11240,11241],{"class":2486},"data_iter",[63,11243,2490],{"class":2483},[63,11245,11246],{"class":2486},"lr",[63,11248,2490],{"class":2483},[63,11250,11251],{"class":2486},"num_epochs",[63,11253,2490],{"class":2483},[63,11255,11256],{"class":2486},"tgt_vocab",[63,11258,2490],{"class":2483},[63,11260,10145],{"class":2486},[63,11262,2496],{"class":2483},[63,11264,11265],{"class":65,"line":88},[63,11266,11267],{"class":2501},"    \"\"\"训练序列到序列模型\"\"\"\n",[63,11269,11270,11272,11275,11277,11279],{"class":65,"line":95},[63,11271,2722],{"class":196},[63,11273,11274],{"class":69}," xavier_init_weights",[63,11276,1078],{"class":2483},[63,11278,643],{"class":2486},[63,11280,2496],{"class":2483},[63,11282,11283,11285,11288,11291,11293],{"class":65,"line":108},[63,11284,2570],{"class":196},[63,11286,11287],{"class":2751}," type",[63,11289,11290],{"class":2483},"(m) ",[63,11292,2576],{"class":196},[63,11294,11295],{"class":2483}," nn.Linear:\n",[63,11297,11298],{"class":65,"line":113},[63,11299,11300],{"class":2483},"            nn.init.xavier_uniform_(m.weight)\n",[63,11302,11303,11305,11307,11309,11311,11314,11317],{"class":65,"line":2548},[63,11304,2570],{"class":196},[63,11306,11287],{"class":2751},[63,11308,11290],{"class":2483},[63,11310,2576],{"class":196},[63,11312,11313],{"class":2483}," nn.",[63,11315,11316],{"class":73},"GRU",[63,11318,2525],{"class":2483},[63,11320,11321,11324,11326,11328],{"class":65,"line":2556},[63,11322,11323],{"class":196},"            for",[63,11325,11179],{"class":2483},[63,11327,9541],{"class":196},[63,11329,11330],{"class":2483}," m._flat_weights_names:\n",[63,11332,11333,11336,11339,11342,11344,11347],{"class":65,"line":2567},[63,11334,11335],{"class":196},"                if",[63,11337,11338],{"class":207}," \"",[63,11340,11341],{"class":77},"weight",[63,11343,208],{"class":207},[63,11345,11346],{"class":196}," in",[63,11348,11349],{"class":2483}," param:\n",[63,11351,11352],{"class":65,"line":2584},[63,11353,11354],{"class":2483},"                    nn.init.xavier_uniform_(m._parameters[param])\n",[63,11356,11357],{"class":65,"line":2600},[63,11358,92],{"emptyLinePlaceholder":91},[63,11360,11361],{"class":65,"line":2608},[63,11362,11363],{"class":2483},"    net.apply(xavier_init_weights)\n",[63,11365,11366],{"class":65,"line":2625},[63,11367,11368],{"class":2483},"    net.to(device)\n",[63,11370,11371,11374,11376,11379,11381,11383],{"class":65,"line":2631},[63,11372,11373],{"class":2483},"    optimizer ",[63,11375,204],{"class":196},[63,11377,11378],{"class":2483}," torch.optim.Adam(net.parameters(), ",[63,11380,11246],{"class":2536},[63,11382,204],{"class":196},[63,11384,11385],{"class":2483},"lr)\n",[63,11387,11388,11391,11393],{"class":65,"line":2656},[63,11389,11390],{"class":2483},"    loss ",[63,11392,204],{"class":196},[63,11394,11395],{"class":2483}," MaskedSoftmaxCELoss()\n",[63,11397,11398],{"class":65,"line":2669},[63,11399,11400],{"class":2483},"    net.train()\n",[63,11402,11403,11406,11408,11411,11414,11416,11418,11421,11423,11425,11428,11430,11432,11435,11437],{"class":65,"line":2907},[63,11404,11405],{"class":2483},"    animator ",[63,11407,204],{"class":196},[63,11409,11410],{"class":2483}," d2l.Animator(",[63,11412,11413],{"class":2536},"xlabel",[63,11415,204],{"class":196},[63,11417,10972],{"class":207},[63,11419,11420],{"class":77},"epoch",[63,11422,10972],{"class":207},[63,11424,2490],{"class":2483},[63,11426,11427],{"class":2536},"ylabel",[63,11429,204],{"class":196},[63,11431,10972],{"class":207},[63,11433,11434],{"class":77},"loss",[63,11436,10972],{"class":207},[63,11438,3891],{"class":2483},[63,11440,11441,11444,11446,11449,11452],{"class":65,"line":4848},[63,11442,11443],{"class":2536},"                     xlim",[63,11445,204],{"class":196},[63,11447,11448],{"class":2483},"[",[63,11450,11451],{"class":73},"10",[63,11453,11454],{"class":2483},", num_epochs])\n",[63,11456,11457,11460,11463,11465,11467],{"class":65,"line":5124},[63,11458,11459],{"class":196},"    for",[63,11461,11462],{"class":2483}," epoch ",[63,11464,9541],{"class":196},[63,11466,9545],{"class":9544},[63,11468,11469],{"class":2483},"(num_epochs):\n",[63,11471,11472,11475,11477],{"class":65,"line":5130},[63,11473,11474],{"class":2483},"        timer ",[63,11476,204],{"class":196},[63,11478,11479],{"class":2483}," d2l.Timer()\n",[63,11481,11482,11485,11487,11490,11492,11495],{"class":65,"line":5136},[63,11483,11484],{"class":2483},"        metric ",[63,11486,204],{"class":196},[63,11488,11489],{"class":2483}," d2l.Accumulator(",[63,11491,2883],{"class":73},[63,11493,11494],{"class":2483},")  ",[63,11496,11497],{"class":2507},"# 训练损失总和，词元数量\n",[63,11499,11500,11502,11505,11507],{"class":65,"line":5142},[63,11501,9535],{"class":196},[63,11503,11504],{"class":2483}," batch ",[63,11506,9541],{"class":196},[63,11508,11509],{"class":2483}," data_iter:\n",[63,11511,11512],{"class":65,"line":5163},[63,11513,11514],{"class":2483},"            optimizer.zero_grad()\n",[63,11516,11517,11520,11522,11525,11527,11530,11532],{"class":65,"line":5182},[63,11518,11519],{"class":2483},"            X, X_valid_len, Y, Y_valid_len ",[63,11521,204],{"class":196},[63,11523,11524],{"class":2483}," [x.to(device) ",[63,11526,10610],{"class":196},[63,11528,11529],{"class":2483}," x ",[63,11531,9541],{"class":196},[63,11533,11534],{"class":2483}," batch]\n",[63,11536,11537,11540,11542,11545,11547,11549,11551,11554,11556,11559,11561],{"class":65,"line":5201},[63,11538,11539],{"class":2483},"            bos ",[63,11541,204],{"class":196},[63,11543,11544],{"class":2483}," torch.tensor([tgt_vocab[",[63,11546,10972],{"class":207},[63,11548,11210],{"class":77},[63,11550,10972],{"class":207},[63,11552,11553],{"class":2483},"]] ",[63,11555,82],{"class":196},[63,11557,11558],{"class":2483}," Y.shape[",[63,11560,1802],{"class":73},[63,11562,11563],{"class":2483},"],\n",[63,11565,11566,11569,11571,11574,11576,11578,11580,11582],{"class":65,"line":5206},[63,11567,11568],{"class":2536},"                          device",[63,11570,204],{"class":196},[63,11572,11573],{"class":2483},"device).reshape(",[63,11575,2618],{"class":196},[63,11577,363],{"class":73},[63,11579,2490],{"class":2483},[63,11581,363],{"class":73},[63,11583,2545],{"class":2483},[63,11585,11586,11589,11591,11594,11596,11598,11600,11602,11604,11607,11609,11611],{"class":65,"line":5222},[63,11587,11588],{"class":2483},"            dec_input ",[63,11590,204],{"class":196},[63,11592,11593],{"class":2483}," torch.cat([bos, Y[",[63,11595,7202],{"class":7201},[63,11597,2490],{"class":2483},[63,11599,7202],{"class":7201},[63,11601,2618],{"class":196},[63,11603,363],{"class":73},[63,11605,11606],{"class":2483},"]], ",[63,11608,363],{"class":73},[63,11610,11494],{"class":2483},[63,11612,11613],{"class":2507},"# 强制教学\n",[63,11615,11616,11619,11621],{"class":65,"line":5228},[63,11617,11618],{"class":2483},"            Y_hat, _ ",[63,11620,204],{"class":196},[63,11622,11623],{"class":2483}," net(X, dec_input, X_valid_len)\n",[63,11625,11626,11629,11631],{"class":65,"line":5234},[63,11627,11628],{"class":2483},"            l ",[63,11630,204],{"class":196},[63,11632,11633],{"class":2483}," loss(Y_hat, Y, Y_valid_len)\n",[63,11635,11636,11639],{"class":65,"line":5244},[63,11637,11638],{"class":2483},"            l.sum().backward()      ",[63,11640,11641],{"class":2507},"# 损失函数的标量进行“反向传播”\n",[63,11643,11644,11647,11649],{"class":65,"line":5268},[63,11645,11646],{"class":2483},"            grad_clipping(net, ",[63,11648,363],{"class":73},[63,11650,2545],{"class":2483},[63,11652,11653,11656,11658],{"class":65,"line":5273},[63,11654,11655],{"class":2483},"            num_tokens ",[63,11657,204],{"class":196},[63,11659,11660],{"class":2483}," Y_valid_len.sum()\n",[63,11662,11663],{"class":65,"line":5279},[63,11664,11665],{"class":2483},"            optimizer.step()\n",[63,11667,11668,11671],{"class":65,"line":5293},[63,11669,11670],{"class":196},"            with",[63,11672,11673],{"class":2483}," torch.no_grad():\n",[63,11675,11676],{"class":65,"line":5298},[63,11677,11678],{"class":2483},"                metric.add(l.sum(), num_tokens)\n",[63,11680,11681,11683,11686,11688,11690,11692,11695,11698,11701,11704],{"class":65,"line":5304},[63,11682,2570],{"class":196},[63,11684,11685],{"class":2483}," (epoch ",[63,11687,5839],{"class":196},[63,11689,2579],{"class":73},[63,11691,7164],{"class":2483},[63,11693,11694],{"class":196},"%",[63,11696,11697],{"class":73}," 10",[63,11699,11700],{"class":196}," ==",[63,11702,11703],{"class":73}," 0",[63,11705,2525],{"class":2483},[63,11707,11708,11711,11713,11715,11718,11720,11722,11724,11727,11729],{"class":65,"line":5319},[63,11709,11710],{"class":2483},"            animator.add(epoch ",[63,11712,5839],{"class":196},[63,11714,2579],{"class":73},[63,11716,11717],{"class":2483},", (metric[",[63,11719,1802],{"class":73},[63,11721,7218],{"class":2483},[63,11723,2889],{"class":196},[63,11725,11726],{"class":2483}," metric[",[63,11728,363],{"class":73},[63,11730,11731],{"class":2483},"],))\n",[63,11733,11734,11737,11739,11742,11745,11748,11751,11753,11755,11757,11759,11761,11764,11767,11770,11772,11774,11776,11778,11780,11782,11785,11788,11790],{"class":65,"line":9624},[63,11735,11736],{"class":9544},"    print",[63,11738,1078],{"class":2483},[63,11740,11741],{"class":196},"f",[63,11743,11744],{"class":77},"'loss ",[63,11746,11747],{"class":73},"{",[63,11749,11750],{"class":2483},"metric[",[63,11752,1802],{"class":73},[63,11754,7218],{"class":2483},[63,11756,2889],{"class":196},[63,11758,11726],{"class":2483},[63,11760,363],{"class":73},[63,11762,11763],{"class":2483},"]",[63,11765,11766],{"class":196},":.3f",[63,11768,11769],{"class":73},"}",[63,11771,2490],{"class":77},[63,11773,11747],{"class":73},[63,11775,11750],{"class":2483},[63,11777,363],{"class":73},[63,11779,7218],{"class":2483},[63,11781,2889],{"class":196},[63,11783,11784],{"class":2483}," timer.stop()",[63,11786,11787],{"class":196},":.1f",[63,11789,11769],{"class":73},[63,11791,11792],{"class":77}," '\n",[63,11794,11795,11798,11801,11803,11805,11808,11810,11812],{"class":65,"line":9630},[63,11796,11797],{"class":196},"        f",[63,11799,11800],{"class":77},"'tokens/sec on ",[63,11802,11747],{"class":73},[63,11804,9568],{"class":2751},[63,11806,11807],{"class":2483},"(device)",[63,11809,11769],{"class":73},[63,11811,10972],{"class":77},[63,11813,2545],{"class":2483},[14,11815,11816],{},"进行数据集的数据处理",[54,11818,11820],{"className":2468,"code":11819,"language":2470,"meta":59,"style":59},"def read_data_nmt():\n    \"\"\"载入“英语－法语”数据集\n\n    Defined in :numref:`sec_machine_translation`\"\"\"\n    data_dir = d2l.download_extract('fra-eng')\n    with open(os.path.join(data_dir, 'fra.txt'), 'r',\n             encoding='utf-8') as f:\n        return f.read()\n\ndef preprocess_nmt(text):\n    \"\"\"预处理“英语－法语”数据集\n\n    Defined in :numref:`sec_machine_translation`\"\"\"\n    def no_space(char, prev_char):\n        return char in set(',.!?') and prev_char != ' '\n\n    # 使用空格替换不间断空格\n    # 使用小写字母替换大写字母\n    text = text.replace('\\u202f', ' ').replace('\\xa0', ' ').lower()\n    # 在单词和标点符号之间插入空格\n    out = [' ' + char if i > 0 and no_space(char, text[i - 1]) else char\n           for i, char in enumerate(text)]\n    return ''.join(out)\n\ndef tokenize_nmt(text, num_examples=None):\n    \"\"\"词元化“英语－法语”数据数据集\n\n    Defined in :numref:`sec_machine_translation`\"\"\"\n    source, target = [], []\n    for i, line in enumerate(text.split('\\n')):\n        if num_examples and i > num_examples:\n            break\n        parts = line.split('\\t')\n        if len(parts) == 2:\n            source.append(parts[0].split(' '))\n            target.append(parts[1].split(' '))\n    return source, target\n\ndef load_data_nmt(batch_size, num_steps, num_examples=600):\n    \"\"\"返回翻译数据集的迭代器和词表\n\n    Defined in :numref:`subsec_mt_data_loading`\"\"\"\n    text = preprocess_nmt(read_data_nmt())\n    source, target = tokenize_nmt(text, num_examples)\n    src_vocab = d2l.Vocab(source, min_freq=2,\n                          reserved_tokens=['\u003Cpad>', '\u003Cbos>', '\u003Ceos>'])\n    tgt_vocab = d2l.Vocab(target, min_freq=2,\n                          reserved_tokens=['\u003Cpad>', '\u003Cbos>', '\u003Ceos>'])\n    src_array, src_valid_len = build_array_nmt(source, src_vocab, num_steps)\n    tgt_array, tgt_valid_len = build_array_nmt(target, tgt_vocab, num_steps)\n    data_arrays = (src_array, src_valid_len, tgt_array, tgt_valid_len)\n    data_iter = d2l.load_array(data_arrays, batch_size)\n    return data_iter, src_vocab, tgt_vocab\n",[49,11821,11822,11832,11837,11841,11846,11865,11894,11916,11923,11927,11940,11945,11949,11953,11972,12009,12013,12018,12023,12066,12071,12116,12131,12141,12145,12167,12172,12176,12180,12190,12214,12230,12235,12254,12269,12286,12301,12308,12312,12340,12345,12349,12354,12363,12372,12391,12425,12443,12475,12485,12495,12505,12515],{"__ignoreMap":59},[63,11823,11824,11826,11829],{"class":65,"line":66},[63,11825,2477],{"class":196},[63,11827,11828],{"class":69}," read_data_nmt",[63,11830,11831],{"class":2483},"():\n",[63,11833,11834],{"class":65,"line":88},[63,11835,11836],{"class":2501},"    \"\"\"载入“英语－法语”数据集\n",[63,11838,11839],{"class":65,"line":95},[63,11840,92],{"emptyLinePlaceholder":91},[63,11842,11843],{"class":65,"line":108},[63,11844,11845],{"class":2501},"    Defined in :numref:`sec_machine_translation`\"\"\"\n",[63,11847,11848,11851,11853,11856,11858,11861,11863],{"class":65,"line":113},[63,11849,11850],{"class":2483},"    data_dir ",[63,11852,204],{"class":196},[63,11854,11855],{"class":2483}," d2l.download_extract(",[63,11857,10972],{"class":207},[63,11859,11860],{"class":77},"fra-eng",[63,11862,10972],{"class":207},[63,11864,2545],{"class":2483},[63,11866,11867,11870,11873,11876,11878,11881,11883,11886,11888,11890,11892],{"class":65,"line":2548},[63,11868,11869],{"class":196},"    with",[63,11871,11872],{"class":9544}," open",[63,11874,11875],{"class":2483},"(os.path.join(data_dir, ",[63,11877,10972],{"class":207},[63,11879,11880],{"class":77},"fra.txt",[63,11882,10972],{"class":207},[63,11884,11885],{"class":2483},"), ",[63,11887,10972],{"class":207},[63,11889,7852],{"class":77},[63,11891,10972],{"class":207},[63,11893,3891],{"class":2483},[63,11895,11896,11899,11901,11903,11906,11908,11910,11913],{"class":65,"line":2556},[63,11897,11898],{"class":2536},"             encoding",[63,11900,204],{"class":196},[63,11902,10972],{"class":207},[63,11904,11905],{"class":77},"utf-8",[63,11907,10972],{"class":207},[63,11909,7164],{"class":2483},[63,11911,11912],{"class":196},"as",[63,11914,11915],{"class":2483}," f:\n",[63,11917,11918,11920],{"class":65,"line":2567},[63,11919,2530],{"class":196},[63,11921,11922],{"class":2483}," f.read()\n",[63,11924,11925],{"class":65,"line":2584},[63,11926,92],{"emptyLinePlaceholder":91},[63,11928,11929,11931,11934,11936,11938],{"class":65,"line":2600},[63,11930,2477],{"class":196},[63,11932,11933],{"class":69}," preprocess_nmt",[63,11935,1078],{"class":2483},[63,11937,5395],{"class":2486},[63,11939,2496],{"class":2483},[63,11941,11942],{"class":65,"line":2608},[63,11943,11944],{"class":2501},"    \"\"\"预处理“英语－法语”数据集\n",[63,11946,11947],{"class":65,"line":2625},[63,11948,92],{"emptyLinePlaceholder":91},[63,11950,11951],{"class":65,"line":2631},[63,11952,11845],{"class":2501},[63,11954,11955,11957,11960,11962,11965,11967,11970],{"class":65,"line":2656},[63,11956,2722],{"class":196},[63,11958,11959],{"class":69}," no_space",[63,11961,1078],{"class":2483},[63,11963,11964],{"class":2486},"char",[63,11966,2490],{"class":2483},[63,11968,11969],{"class":2486},"prev_char",[63,11971,2496],{"class":2483},[63,11973,11974,11976,11979,11981,11984,11986,11988,11991,11993,11995,11998,12001,12004,12007],{"class":65,"line":2669},[63,11975,2530],{"class":196},[63,11977,11978],{"class":2483}," char ",[63,11980,9541],{"class":196},[63,11982,11983],{"class":2751}," set",[63,11985,1078],{"class":2483},[63,11987,10972],{"class":207},[63,11989,11990],{"class":77},",.!?",[63,11992,10972],{"class":207},[63,11994,7164],{"class":2483},[63,11996,11997],{"class":196},"and",[63,11999,12000],{"class":2483}," prev_char ",[63,12002,12003],{"class":196},"!=",[63,12005,12006],{"class":207}," '",[63,12008,11792],{"class":207},[63,12010,12011],{"class":65,"line":2907},[63,12012,92],{"emptyLinePlaceholder":91},[63,12014,12015],{"class":65,"line":4848},[63,12016,12017],{"class":2507},"    # 使用空格替换不间断空格\n",[63,12019,12020],{"class":65,"line":5124},[63,12021,12022],{"class":2507},"    # 使用小写字母替换大写字母\n",[63,12024,12025,12028,12030,12033,12035,12039,12041,12043,12045,12047,12050,12052,12055,12057,12059,12061,12063],{"class":65,"line":5130},[63,12026,12027],{"class":2483},"    text ",[63,12029,204],{"class":196},[63,12031,12032],{"class":2483}," text.replace(",[63,12034,10972],{"class":207},[63,12036,12038],{"class":12037},"sRfyP","\\u202f",[63,12040,10972],{"class":207},[63,12042,2490],{"class":2483},[63,12044,10972],{"class":207},[63,12046,12006],{"class":207},[63,12048,12049],{"class":2483},").replace(",[63,12051,10972],{"class":207},[63,12053,12054],{"class":12037},"\\xa0",[63,12056,10972],{"class":207},[63,12058,2490],{"class":2483},[63,12060,10972],{"class":207},[63,12062,12006],{"class":207},[63,12064,12065],{"class":2483},").lower()\n",[63,12067,12068],{"class":65,"line":5136},[63,12069,12070],{"class":2507},"    # 在单词和标点符号之间插入空格\n",[63,12072,12073,12076,12078,12080,12082,12084,12087,12089,12091,12093,12095,12097,12100,12103,12105,12107,12110,12113],{"class":65,"line":5142},[63,12074,12075],{"class":2483},"    out ",[63,12077,204],{"class":196},[63,12079,9672],{"class":2483},[63,12081,10972],{"class":207},[63,12083,12006],{"class":207},[63,12085,12086],{"class":196}," +",[63,12088,11978],{"class":2483},[63,12090,11109],{"class":196},[63,12092,9538],{"class":2483},[63,12094,11169],{"class":196},[63,12096,11703],{"class":73},[63,12098,12099],{"class":196}," and",[63,12101,12102],{"class":2483}," no_space(char, text[i ",[63,12104,2618],{"class":196},[63,12106,2579],{"class":73},[63,12108,12109],{"class":2483},"]) ",[63,12111,12112],{"class":196},"else",[63,12114,12115],{"class":2483}," char\n",[63,12117,12118,12121,12124,12126,12128],{"class":65,"line":5163},[63,12119,12120],{"class":196},"           for",[63,12122,12123],{"class":2483}," i, char ",[63,12125,9541],{"class":196},[63,12127,9701],{"class":9544},[63,12129,12130],{"class":2483},"(text)]\n",[63,12132,12133,12135,12138],{"class":65,"line":5182},[63,12134,4754],{"class":196},[63,12136,12137],{"class":207}," ''",[63,12139,12140],{"class":2483},".join(out)\n",[63,12142,12143],{"class":65,"line":5201},[63,12144,92],{"emptyLinePlaceholder":91},[63,12146,12147,12149,12152,12154,12156,12158,12161,12163,12165],{"class":65,"line":5206},[63,12148,2477],{"class":196},[63,12150,12151],{"class":69}," tokenize_nmt",[63,12153,1078],{"class":2483},[63,12155,5395],{"class":2486},[63,12157,2490],{"class":2483},[63,12159,12160],{"class":2486},"num_examples",[63,12162,204],{"class":196},[63,12164,2842],{"class":73},[63,12166,2496],{"class":2483},[63,12168,12169],{"class":65,"line":5222},[63,12170,12171],{"class":2501},"    \"\"\"词元化“英语－法语”数据数据集\n",[63,12173,12174],{"class":65,"line":5228},[63,12175,92],{"emptyLinePlaceholder":91},[63,12177,12178],{"class":65,"line":5234},[63,12179,11845],{"class":2501},[63,12181,12182,12185,12187],{"class":65,"line":5244},[63,12183,12184],{"class":2483},"    source, target ",[63,12186,204],{"class":196},[63,12188,12189],{"class":2483}," [], []\n",[63,12191,12192,12194,12197,12199,12201,12204,12206,12209,12211],{"class":65,"line":5268},[63,12193,11459],{"class":196},[63,12195,12196],{"class":2483}," i, line ",[63,12198,9541],{"class":196},[63,12200,9701],{"class":9544},[63,12202,12203],{"class":2483},"(text.split(",[63,12205,10972],{"class":207},[63,12207,12208],{"class":12037},"\\n",[63,12210,10972],{"class":207},[63,12212,12213],{"class":2483},")):\n",[63,12215,12216,12218,12221,12223,12225,12227],{"class":65,"line":5273},[63,12217,2570],{"class":196},[63,12219,12220],{"class":2483}," num_examples ",[63,12222,11997],{"class":196},[63,12224,9538],{"class":2483},[63,12226,11169],{"class":196},[63,12228,12229],{"class":2483}," num_examples:\n",[63,12231,12232],{"class":65,"line":5279},[63,12233,12234],{"class":196},"            break\n",[63,12236,12237,12240,12242,12245,12247,12250,12252],{"class":65,"line":5293},[63,12238,12239],{"class":2483},"        parts ",[63,12241,204],{"class":196},[63,12243,12244],{"class":2483}," line.split(",[63,12246,10972],{"class":207},[63,12248,12249],{"class":12037},"\\t",[63,12251,10972],{"class":207},[63,12253,2545],{"class":2483},[63,12255,12256,12258,12260,12263,12265,12267],{"class":65,"line":5298},[63,12257,2570],{"class":196},[63,12259,9681],{"class":9544},[63,12261,12262],{"class":2483},"(parts) ",[63,12264,2576],{"class":196},[63,12266,11148],{"class":73},[63,12268,2525],{"class":2483},[63,12270,12271,12274,12276,12279,12281,12283],{"class":65,"line":5304},[63,12272,12273],{"class":2483},"            source.append(parts[",[63,12275,1802],{"class":73},[63,12277,12278],{"class":2483},"].split(",[63,12280,10972],{"class":207},[63,12282,12006],{"class":207},[63,12284,12285],{"class":2483},"))\n",[63,12287,12288,12291,12293,12295,12297,12299],{"class":65,"line":5319},[63,12289,12290],{"class":2483},"            target.append(parts[",[63,12292,363],{"class":73},[63,12294,12278],{"class":2483},[63,12296,10972],{"class":207},[63,12298,12006],{"class":207},[63,12300,12285],{"class":2483},[63,12302,12303,12305],{"class":65,"line":9624},[63,12304,4754],{"class":196},[63,12306,12307],{"class":2483}," source, target\n",[63,12309,12310],{"class":65,"line":9630},[63,12311,92],{"emptyLinePlaceholder":91},[63,12313,12314,12316,12319,12321,12324,12326,12329,12331,12333,12335,12338],{"class":65,"line":9636},[63,12315,2477],{"class":196},[63,12317,12318],{"class":69}," load_data_nmt",[63,12320,1078],{"class":2483},[63,12322,12323],{"class":2486},"batch_size",[63,12325,2490],{"class":2483},[63,12327,12328],{"class":2486},"num_steps",[63,12330,2490],{"class":2483},[63,12332,12160],{"class":2486},[63,12334,204],{"class":196},[63,12336,12337],{"class":73},"600",[63,12339,2496],{"class":2483},[63,12341,12342],{"class":65,"line":9663},[63,12343,12344],{"class":2501},"    \"\"\"返回翻译数据集的迭代器和词表\n",[63,12346,12347],{"class":65,"line":9691},[63,12348,92],{"emptyLinePlaceholder":91},[63,12350,12351],{"class":65,"line":9711},[63,12352,12353],{"class":2501},"    Defined in :numref:`subsec_mt_data_loading`\"\"\"\n",[63,12355,12356,12358,12360],{"class":65,"line":9722},[63,12357,12027],{"class":2483},[63,12359,204],{"class":196},[63,12361,12362],{"class":2483}," preprocess_nmt(read_data_nmt())\n",[63,12364,12365,12367,12369],{"class":65,"line":9735},[63,12366,12184],{"class":2483},[63,12368,204],{"class":196},[63,12370,12371],{"class":2483}," tokenize_nmt(text, num_examples)\n",[63,12373,12374,12377,12379,12382,12385,12387,12389],{"class":65,"line":10238},[63,12375,12376],{"class":2483},"    src_vocab ",[63,12378,204],{"class":196},[63,12380,12381],{"class":2483}," d2l.Vocab(source, ",[63,12383,12384],{"class":2536},"min_freq",[63,12386,204],{"class":196},[63,12388,2883],{"class":73},[63,12390,3891],{"class":2483},[63,12392,12393,12396,12398,12400,12402,12405,12407,12409,12411,12413,12415,12417,12419,12421,12423],{"class":65,"line":10253},[63,12394,12395],{"class":2536},"                          reserved_tokens",[63,12397,204],{"class":196},[63,12399,11448],{"class":2483},[63,12401,10972],{"class":207},[63,12403,12404],{"class":77},"\u003Cpad>",[63,12406,10972],{"class":207},[63,12408,2490],{"class":2483},[63,12410,10972],{"class":207},[63,12412,11210],{"class":77},[63,12414,10972],{"class":207},[63,12416,2490],{"class":2483},[63,12418,10972],{"class":207},[63,12420,11214],{"class":77},[63,12422,10972],{"class":207},[63,12424,2597],{"class":2483},[63,12426,12427,12430,12432,12435,12437,12439,12441],{"class":65,"line":10258},[63,12428,12429],{"class":2483},"    tgt_vocab ",[63,12431,204],{"class":196},[63,12433,12434],{"class":2483}," d2l.Vocab(target, ",[63,12436,12384],{"class":2536},[63,12438,204],{"class":196},[63,12440,2883],{"class":73},[63,12442,3891],{"class":2483},[63,12444,12445,12447,12449,12451,12453,12455,12457,12459,12461,12463,12465,12467,12469,12471,12473],{"class":65,"line":10273},[63,12446,12395],{"class":2536},[63,12448,204],{"class":196},[63,12450,11448],{"class":2483},[63,12452,10972],{"class":207},[63,12454,12404],{"class":77},[63,12456,10972],{"class":207},[63,12458,2490],{"class":2483},[63,12460,10972],{"class":207},[63,12462,11210],{"class":77},[63,12464,10972],{"class":207},[63,12466,2490],{"class":2483},[63,12468,10972],{"class":207},[63,12470,11214],{"class":77},[63,12472,10972],{"class":207},[63,12474,2597],{"class":2483},[63,12476,12477,12480,12482],{"class":65,"line":10302},[63,12478,12479],{"class":2483},"    src_array, src_valid_len ",[63,12481,204],{"class":196},[63,12483,12484],{"class":2483}," build_array_nmt(source, src_vocab, num_steps)\n",[63,12486,12487,12490,12492],{"class":65,"line":10321},[63,12488,12489],{"class":2483},"    tgt_array, tgt_valid_len ",[63,12491,204],{"class":196},[63,12493,12494],{"class":2483}," build_array_nmt(target, tgt_vocab, num_steps)\n",[63,12496,12497,12500,12502],{"class":65,"line":10342},[63,12498,12499],{"class":2483},"    data_arrays ",[63,12501,204],{"class":196},[63,12503,12504],{"class":2483}," (src_array, src_valid_len, tgt_array, tgt_valid_len)\n",[63,12506,12507,12510,12512],{"class":65,"line":10362},[63,12508,12509],{"class":2483},"    data_iter ",[63,12511,204],{"class":196},[63,12513,12514],{"class":2483}," d2l.load_array(data_arrays, batch_size)\n",[63,12516,12517,12519],{"class":65,"line":10373},[63,12518,4754],{"class":196},[63,12520,12521],{"class":2483}," data_iter, src_vocab, tgt_vocab\n",[14,12523,12524],{},"完成数据处理后，按照 Transformer 架构搭建编码器-解码器模型并启动训练：",[54,12526,12528],{"className":2468,"code":12527,"language":2470,"meta":59,"style":59},"num_hiddens, num_layers, dropout, batch_size, num_steps = 32, 2, 0.1, 64, 10\nlr, num_epochs, device = 0.005, 200, torch.device('cuda:0') if torch.cuda.device_count() >= 1 else torch.device('cpu')\nffn_num_hiddens, num_heads = 64, 4\n\ntrain_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)\n\nencoder = TransformerEncoder(\n    len(src_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_layers,\n    dropout)\ndecoder = TransformerDecoder(\n    len(tgt_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_layers,\n    dropout)\nnet = EncoderDecoder(encoder, decoder)\ntrain_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)\n",[49,12529,12530,12559,12611,12626,12630,12640,12644,12654,12662,12667,12677,12684,12688,12698],{"__ignoreMap":59},[63,12531,12532,12535,12537,12540,12542,12544,12546,12549,12551,12554,12556],{"class":65,"line":66},[63,12533,12534],{"class":2483},"num_hiddens, num_layers, dropout, batch_size, num_steps ",[63,12536,204],{"class":196},[63,12538,12539],{"class":73}," 32",[63,12541,2490],{"class":2483},[63,12543,2883],{"class":73},[63,12545,2490],{"class":2483},[63,12547,12548],{"class":73},"0.1",[63,12550,2490],{"class":2483},[63,12552,12553],{"class":73},"64",[63,12555,2490],{"class":2483},[63,12557,12558],{"class":73},"10\n",[63,12560,12561,12564,12566,12569,12571,12574,12577,12579,12582,12584,12586,12588,12591,12594,12596,12599,12602,12604,12607,12609],{"class":65,"line":88},[63,12562,12563],{"class":2483},"lr, num_epochs, device ",[63,12565,204],{"class":196},[63,12567,12568],{"class":73}," 0.005",[63,12570,2490],{"class":2483},[63,12572,12573],{"class":73},"200",[63,12575,12576],{"class":2483},", torch.device(",[63,12578,10972],{"class":207},[63,12580,12581],{"class":77},"cuda:0",[63,12583,10972],{"class":207},[63,12585,7164],{"class":2483},[63,12587,11109],{"class":196},[63,12589,12590],{"class":2483}," torch.cuda.device_count() ",[63,12592,12593],{"class":196},">=",[63,12595,2579],{"class":73},[63,12597,12598],{"class":196}," else",[63,12600,12601],{"class":2483}," torch.device(",[63,12603,10972],{"class":207},[63,12605,12606],{"class":77},"cpu",[63,12608,10972],{"class":207},[63,12610,2545],{"class":2483},[63,12612,12613,12616,12618,12621,12623],{"class":65,"line":95},[63,12614,12615],{"class":2483},"ffn_num_hiddens, num_heads ",[63,12617,204],{"class":196},[63,12619,12620],{"class":73}," 64",[63,12622,2490],{"class":2483},[63,12624,12625],{"class":73},"4\n",[63,12627,12628],{"class":65,"line":108},[63,12629,92],{"emptyLinePlaceholder":91},[63,12631,12632,12635,12637],{"class":65,"line":113},[63,12633,12634],{"class":2483},"train_iter, src_vocab, tgt_vocab ",[63,12636,204],{"class":196},[63,12638,12639],{"class":2483}," d2l.load_data_nmt(batch_size, num_steps)\n",[63,12641,12642],{"class":65,"line":2548},[63,12643,92],{"emptyLinePlaceholder":91},[63,12645,12646,12649,12651],{"class":65,"line":2556},[63,12647,12648],{"class":2483},"encoder ",[63,12650,204],{"class":196},[63,12652,12653],{"class":2483}," TransformerEncoder(\n",[63,12655,12656,12659],{"class":65,"line":2567},[63,12657,12658],{"class":9544},"    len",[63,12660,12661],{"class":2483},"(src_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_layers,\n",[63,12663,12664],{"class":65,"line":2584},[63,12665,12666],{"class":2483},"    dropout)\n",[63,12668,12669,12672,12674],{"class":65,"line":2600},[63,12670,12671],{"class":2483},"decoder ",[63,12673,204],{"class":196},[63,12675,12676],{"class":2483}," TransformerDecoder(\n",[63,12678,12679,12681],{"class":65,"line":2608},[63,12680,12658],{"class":9544},[63,12682,12683],{"class":2483},"(tgt_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_layers,\n",[63,12685,12686],{"class":65,"line":2625},[63,12687,12666],{"class":2483},[63,12689,12690,12693,12695],{"class":65,"line":2631},[63,12691,12692],{"class":2483},"net ",[63,12694,204],{"class":196},[63,12696,12697],{"class":2483}," EncoderDecoder(encoder, decoder)\n",[63,12699,12700],{"class":65,"line":2656},[63,12701,12702],{"class":2483},"train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)\n",[10,12704,12705],{"id":12705},"参考文献",[14,12707,12708],{},[38,12709,316],{"href":314,"rel":12710},[42],[14,12712,12713],{},[38,12714,310],{"href":308,"rel":12715},[42],[266,12717,12718],{},"html pre.shiki code .szJfE, html code.shiki .szJfE{--shiki-default:#D73A49;--shiki-dark:#FF79C6}html pre.shiki code .sCLZk, html code.shiki .sCLZk{--shiki-default:#6F42C1;--shiki-dark:#50FA7B}html pre.shiki code .scbbO, html code.shiki .scbbO{--shiki-default:#24292E;--shiki-dark:#F8F8F2}html pre.shiki code .syNf4, html code.shiki .syNf4{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#FFB86C;--shiki-dark-font-style:italic}html pre.shiki code .seLWX, html code.shiki .seLWX{--shiki-default:#032F62;--shiki-dark:#6272A4}html pre.shiki code .sfgPZ, html code.shiki .sfgPZ{--shiki-default:#6A737D;--shiki-dark:#6272A4}html pre.shiki code .soDru, html code.shiki .soDru{--shiki-default:#005CC5;--shiki-dark:#BD93F9}html pre.shiki code .sQkXh, html code.shiki .sQkXh{--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFB86C;--shiki-dark-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .skCyd, html code.shiki .skCyd{--shiki-default:#6F42C1;--shiki-dark:#8BE9FD}html pre.shiki code .sDP9b, html code.shiki .sDP9b{--shiki-default:#6F42C1;--shiki-default-font-style:inherit;--shiki-dark:#8BE9FD;--shiki-dark-font-style:italic}html pre.shiki code .sD3jg, html code.shiki .sD3jg{--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#BD93F9;--shiki-dark-font-style:italic}html pre.shiki code .sPGBF, html code.shiki .sPGBF{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#8BE9FD;--shiki-dark-font-style:italic}html pre.shiki code .sJti5, html code.shiki .sJti5{--shiki-default:#005CC5;--shiki-default-font-style:inherit;--shiki-dark:#BD93F9;--shiki-dark-font-style:italic}html pre.shiki code .sDoOe, html code.shiki .sDoOe{--shiki-default:#24292E;--shiki-dark:#FF79C6}html pre.shiki code .sDgm9, html code.shiki .sDgm9{--shiki-default:#005CC5;--shiki-dark:#8BE9FD}html pre.shiki code .sMWOi, html code.shiki .sMWOi{--shiki-default:#032F62;--shiki-dark:#E9F284}html pre.shiki code .sEzAm, html code.shiki .sEzAm{--shiki-default:#032F62;--shiki-dark:#F1FA8C}html pre.shiki code .sRfyP, html code.shiki .sRfyP{--shiki-default:#005CC5;--shiki-dark:#FF79C6}",{"title":59,"searchDepth":88,"depth":88,"links":12720},[12721,12722,12723,12727,12728,12729,12730,12731,12732,12733,12734],{"id":12,"depth":88,"text":12},{"id":319,"depth":88,"text":319},{"id":329,"depth":88,"text":329,"children":12724},[12725,12726],{"id":332,"depth":95,"text":332},{"id":2936,"depth":95,"text":2936},{"id":5329,"depth":88,"text":5329},{"id":7315,"depth":88,"text":7315},{"id":7819,"depth":88,"text":7819},{"id":8213,"depth":88,"text":8213},{"id":9127,"depth":88,"text":9128},{"id":9743,"depth":88,"text":9744},{"id":10750,"depth":88,"text":10750},{"id":12705,"depth":88,"text":12705},"距离首次学习 transformer 已经过去一年，内容忘的差不多了，决定复习一下。",{"date":12737,"image":12738,"alt":295,"tags":12739,"published":91},"30st Jan 2026","/blogs-img/blog1.jpg",[12740,12741],"deep-learning","学习笔记","/blogs/transformer",{"title":295,"description":12735},"blogs/1. transformer复习笔记","1ycHTU2X8OkVcaQ6csyYCF8ZjnFmnQXik5F9B9P0AtI",{"id":12747,"title":12748,"body":12749,"description":45437,"extension":280,"meta":45438,"navigation":91,"ogImage":45440,"path":45444,"seo":45445,"stem":45446,"__hash__":45447},"content/blogs/2. PPO学习笔记.md","零基础速通 PPO 学习笔记",{"type":7,"value":12750,"toc":45424},[12751,12754,12757,12760,12763,13304,13607,14071,14077,14080,14155,14492,18485,19087,19606,19610,19773,20435,20801,21134,21137,21823,21826,22965,23138,23211,25842,25965,27836,27840,27843,27937,30718,30721,30829,30832,32319,32392,32396,32590,32593,32931,34595,34598,34601,35253,35444,39969,39972,41017,41338,41343,41408,42699,42703,42713,43263,43427,45411,45414,45417],[10,12752,12753],{"id":12753},"学习背景",[14,12755,12756],{},"最近套磁到了港科的一个 RA 岗位，组里在做 Agent 强化学习相关的研究，按照学长的建议先把 PPO 算法系统学习一遍，于是有了这篇笔记。",[10,12758,12759],{"id":12759},"符号定义",[14,12761,12762],{},"在强化学习中，常用符号约定如下：",[12764,12765,12766,12872,12974,13156,13258,13292,13298],"ul",{},[12767,12768,12769,12841,12842,12871],"li",{},[63,12770,12772,12790],{"className":12771},[342],[63,12773,12775],{"className":12774},[346],[348,12776,12777],{"xmlns":350},[352,12778,12779,12787],{},[355,12780,12781],{},[368,12782,12783,12785],{},[371,12784,38],{},[371,12786,1057],{},[378,12788,12789],{"encoding":380},"a_t",[63,12791,12793],{"className":12792,"ariaHidden":386},[385],[63,12794,12796,12800],{"className":12795},[390],[63,12797],{"className":12798,"style":12799},[394],"height:0.5806em;vertical-align:-0.15em;",[63,12801,12803,12806],{"className":12802},[399],[63,12804,38],{"className":12805},[399,472],[63,12807,12809],{"className":12808},[476],[63,12810,12812,12833],{"className":12811},[411,412],[63,12813,12815,12830],{"className":12814},[416],[63,12816,12819],{"className":12817,"style":12818},[420],"height:0.2806em;",[63,12820,12821,12824],{"style":1535},[63,12822],{"className":12823,"style":817},[428],[63,12825,12827],{"className":12826},[433,434,435,436],[63,12828,1057],{"className":12829},[399,472,436],[63,12831,507],{"className":12832},[506],[63,12834,12836],{"className":12835},[416],[63,12837,12839],{"className":12838,"style":1554},[420],[63,12840],{},"：",[63,12843,12845,12858],{"className":12844},[342],[63,12846,12848],{"className":12847},[346],[348,12849,12850],{"xmlns":350},[352,12851,12852,12856],{},[355,12853,12854],{},[371,12855,1057],{},[378,12857,1057],{"encoding":380},[63,12859,12861],{"className":12860,"ariaHidden":386},[385],[63,12862,12864,12868],{"className":12863},[390],[63,12865],{"className":12866,"style":12867},[394],"height:0.6151em;",[63,12869,1057],{"className":12870},[399,472]," 时刻 Agent 采取的动作（action）；",[12767,12873,12874,12841,12945,12973],{},[63,12875,12877,12896],{"className":12876},[342],[63,12878,12880],{"className":12879},[346],[348,12881,12882],{"xmlns":350},[352,12883,12884,12893],{},[355,12885,12886],{},[368,12887,12888,12891],{},[371,12889,12890],{},"s",[371,12892,1057],{},[378,12894,12895],{"encoding":380},"s_t",[63,12897,12899],{"className":12898,"ariaHidden":386},[385],[63,12900,12902,12905],{"className":12901},[390],[63,12903],{"className":12904,"style":12799},[394],[63,12906,12908,12911],{"className":12907},[399],[63,12909,12890],{"className":12910},[399,472],[63,12912,12914],{"className":12913},[476],[63,12915,12917,12937],{"className":12916},[411,412],[63,12918,12920,12934],{"className":12919},[416],[63,12921,12923],{"className":12922,"style":12818},[420],[63,12924,12925,12928],{"style":1535},[63,12926],{"className":12927,"style":817},[428],[63,12929,12931],{"className":12930},[433,434,435,436],[63,12932,1057],{"className":12933},[399,472,436],[63,12935,507],{"className":12936},[506],[63,12938,12940],{"className":12939},[416],[63,12941,12943],{"className":12942,"style":1554},[420],[63,12944],{},[63,12946,12948,12961],{"className":12947},[342],[63,12949,12951],{"className":12950},[346],[348,12952,12953],{"xmlns":350},[352,12954,12955,12959],{},[355,12956,12957],{},[371,12958,1057],{},[378,12960,1057],{"encoding":380},[63,12962,12964],{"className":12963,"ariaHidden":386},[385],[63,12965,12967,12970],{"className":12966},[390],[63,12968],{"className":12969,"style":12867},[394],[63,12971,1057],{"className":12972},[399,472]," 时刻 Agent 所处的状态（state）；",[12767,12975,12976,13005,13006,13155],{},[63,12977,12979,12993],{"className":12978},[342],[63,12980,12982],{"className":12981},[346],[348,12983,12984],{"xmlns":350},[352,12985,12986,12990],{},[355,12987,12988],{},[371,12989,6764],{},[378,12991,12992],{"encoding":380},"\\pi",[63,12994,12996],{"className":12995,"ariaHidden":386},[385],[63,12997,12999,13002],{"className":12998},[390],[63,13000],{"className":13001,"style":624},[394],[63,13003,6764],{"className":13004,"style":718},[399,472],"：Agent 的策略函数（policy），输入状态，输出每个动作的概率分布 ",[63,13007,13009,13042],{"className":13008},[342],[63,13010,13012],{"className":13011},[346],[348,13013,13014],{"xmlns":350},[352,13015,13016,13039],{},[355,13017,13018,13020,13022,13028,13031,13037],{},[371,13019,6764],{},[738,13021,1078],{"stretchy":1077},[368,13023,13024,13026],{},[371,13025,38],{},[371,13027,1057],{},[738,13029,13030],{},"∣",[368,13032,13033,13035],{},[371,13034,12890],{},[371,13036,1057],{},[738,13038,1092],{"stretchy":1077},[378,13040,13041],{"encoding":380},"\\pi(a_t \\mid s_t)",[63,13043,13045,13106],{"className":13044,"ariaHidden":386},[385],[63,13046,13048,13051,13054,13057,13097,13100,13103],{"className":13047},[390],[63,13049],{"className":13050,"style":1155},[394],[63,13052,6764],{"className":13053,"style":718},[399,472],[63,13055,1078],{"className":13056},[403],[63,13058,13060,13063],{"className":13059},[399],[63,13061,38],{"className":13062},[399,472],[63,13064,13066],{"className":13065},[476],[63,13067,13069,13089],{"className":13068},[411,412],[63,13070,13072,13086],{"className":13071},[416],[63,13073,13075],{"className":13074,"style":12818},[420],[63,13076,13077,13080],{"style":1535},[63,13078],{"className":13079,"style":817},[428],[63,13081,13083],{"className":13082},[433,434,435,436],[63,13084,1057],{"className":13085},[399,472,436],[63,13087,507],{"className":13088},[506],[63,13090,13092],{"className":13091},[416],[63,13093,13095],{"className":13094,"style":1554},[420],[63,13096],{},[63,13098],{"className":13099,"style":777},[776],[63,13101,13030],{"className":13102},[781],[63,13104],{"className":13105,"style":777},[776],[63,13107,13109,13112,13152],{"className":13108},[390],[63,13110],{"className":13111,"style":1155},[394],[63,13113,13115,13118],{"className":13114},[399],[63,13116,12890],{"className":13117},[399,472],[63,13119,13121],{"className":13120},[476],[63,13122,13124,13144],{"className":13123},[411,412],[63,13125,13127,13141],{"className":13126},[416],[63,13128,13130],{"className":13129,"style":12818},[420],[63,13131,13132,13135],{"style":1535},[63,13133],{"className":13134,"style":817},[428],[63,13136,13138],{"className":13137},[433,434,435,436],[63,13139,1057],{"className":13140},[399,472,436],[63,13142,507],{"className":13143},[506],[63,13145,13147],{"className":13146},[416],[63,13148,13150],{"className":13149,"style":1554},[420],[63,13151],{},[63,13153,1092],{"className":13154},[593],"；",[12767,13157,13158,12841,13229,13257],{},[63,13159,13161,13179],{"className":13160},[342],[63,13162,13164],{"className":13163},[346],[348,13165,13166],{"xmlns":350},[352,13167,13168,13176],{},[355,13169,13170],{},[368,13171,13172,13174],{},[371,13173,7852],{},[371,13175,1057],{},[378,13177,13178],{"encoding":380},"r_t",[63,13180,13182],{"className":13181,"ariaHidden":386},[385],[63,13183,13185,13188],{"className":13184},[390],[63,13186],{"className":13187,"style":12799},[394],[63,13189,13191,13194],{"className":13190},[399],[63,13192,7852],{"className":13193,"style":3276},[399,472],[63,13195,13197],{"className":13196},[476],[63,13198,13200,13221],{"className":13199},[411,412],[63,13201,13203,13218],{"className":13202},[416],[63,13204,13206],{"className":13205,"style":12818},[420],[63,13207,13209,13212],{"style":13208},"top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;",[63,13210],{"className":13211,"style":817},[428],[63,13213,13215],{"className":13214},[433,434,435,436],[63,13216,1057],{"className":13217},[399,472,436],[63,13219,507],{"className":13220},[506],[63,13222,13224],{"className":13223},[416],[63,13225,13227],{"className":13226,"style":1554},[420],[63,13228],{},[63,13230,13232,13245],{"className":13231},[342],[63,13233,13235],{"className":13234},[346],[348,13236,13237],{"xmlns":350},[352,13238,13239,13243],{},[355,13240,13241],{},[371,13242,1057],{},[378,13244,1057],{"encoding":380},[63,13246,13248],{"className":13247,"ariaHidden":386},[385],[63,13249,13251,13254],{"className":13250},[390],[63,13252],{"className":13253,"style":12867},[394],[63,13255,1057],{"className":13256},[399,472]," 时刻 Agent 采取动作后获得的奖励（reward）；",[12767,13259,13260,13291],{},[63,13261,13263,13278],{"className":13262},[342],[63,13264,13266],{"className":13265},[346],[348,13267,13268],{"xmlns":350},[352,13269,13270,13275],{},[355,13271,13272],{},[371,13273,13274],{},"τ",[378,13276,13277],{"encoding":380},"\\tau",[63,13279,13281],{"className":13280,"ariaHidden":386},[385],[63,13282,13284,13287],{"className":13283},[390],[63,13285],{"className":13286,"style":624},[394],[63,13288,13274],{"className":13289,"style":13290},[399,472],"margin-right:0.1132em;","：一条轨迹（trajectory），即一段状态-动作序列；",[12767,13293,13294,13297],{},[11217,13295,13296],{},"Episode","：一次完整的交互过程，从环境初始化开始，到达到终止状态为止；",[12767,13299,13300,13303],{},[11217,13301,13302],{},"Rollout","：按当前策略实际\"跑出来\"的一段轨迹数据，不一定是完整的一局，也可以只是其中的一段。",[14,13305,13306],{},[63,13307,13309,13364],{"className":13308},[342],[63,13310,13312],{"className":13311},[346],[348,13313,13314],{"xmlns":350},[352,13315,13316,13361],{},[355,13317,13318,13320,13326,13328,13334,13336,13342,13344,13350,13352,13359],{},[738,13319,1078],{"stretchy":1077},[368,13321,13322,13324],{},[371,13323,12890],{},[361,13325,1802],{},[738,13327,1083],{"separator":386},[368,13329,13330,13332],{},[371,13331,38],{},[361,13333,1802],{},[738,13335,1083],{"separator":386},[368,13337,13338,13340],{},[371,13339,12890],{},[361,13341,363],{},[738,13343,1083],{"separator":386},[368,13345,13346,13348],{},[371,13347,38],{},[361,13349,363],{},[738,13351,3036],{},[368,13353,13354,13356],{},[371,13355,12890],{},[371,13357,13358],{},"T",[738,13360,1092],{"stretchy":1077},[378,13362,13363],{"encoding":380},"(s_0,a_0,s_1,a_1 \\dots s_T)",[63,13365,13367],{"className":13366,"ariaHidden":386},[385],[63,13368,13370,13373,13376,13416,13419,13422,13462,13465,13468,13508,13511,13514,13554,13557,13560,13563,13604],{"className":13369},[390],[63,13371],{"className":13372,"style":1155},[394],[63,13374,1078],{"className":13375},[403],[63,13377,13379,13382],{"className":13378},[399],[63,13380,12890],{"className":13381},[399,472],[63,13383,13385],{"className":13384},[476],[63,13386,13388,13408],{"className":13387},[411,412],[63,13389,13391,13405],{"className":13390},[416],[63,13392,13394],{"className":13393,"style":3157},[420],[63,13395,13396,13399],{"style":1535},[63,13397],{"className":13398,"style":817},[428],[63,13400,13402],{"className":13401},[433,434,435,436],[63,13403,1802],{"className":13404},[399,436],[63,13406,507],{"className":13407},[506],[63,13409,13411],{"className":13410},[416],[63,13412,13414],{"className":13413,"style":1554},[420],[63,13415],{},[63,13417,1083],{"className":13418},[1173],[63,13420],{"className":13421,"style":1177},[776],[63,13423,13425,13428],{"className":13424},[399],[63,13426,38],{"className":13427},[399,472],[63,13429,13431],{"className":13430},[476],[63,13432,13434,13454],{"className":13433},[411,412],[63,13435,13437,13451],{"className":13436},[416],[63,13438,13440],{"className":13439,"style":3157},[420],[63,13441,13442,13445],{"style":1535},[63,13443],{"className":13444,"style":817},[428],[63,13446,13448],{"className":13447},[433,434,435,436],[63,13449,1802],{"className":13450},[399,436],[63,13452,507],{"className":13453},[506],[63,13455,13457],{"className":13456},[416],[63,13458,13460],{"className":13459,"style":1554},[420],[63,13461],{},[63,13463,1083],{"className":13464},[1173],[63,13466],{"className":13467,"style":1177},[776],[63,13469,13471,13474],{"className":13470},[399],[63,13472,12890],{"className":13473},[399,472],[63,13475,13477],{"className":13476},[476],[63,13478,13480,13500],{"className":13479},[411,412],[63,13481,13483,13497],{"className":13482},[416],[63,13484,13486],{"className":13485,"style":3157},[420],[63,13487,13488,13491],{"style":1535},[63,13489],{"className":13490,"style":817},[428],[63,13492,13494],{"className":13493},[433,434,435,436],[63,13495,363],{"className":13496},[399,436],[63,13498,507],{"className":13499},[506],[63,13501,13503],{"className":13502},[416],[63,13504,13506],{"className":13505,"style":1554},[420],[63,13507],{},[63,13509,1083],{"className":13510},[1173],[63,13512],{"className":13513,"style":1177},[776],[63,13515,13517,13520],{"className":13516},[399],[63,13518,38],{"className":13519},[399,472],[63,13521,13523],{"className":13522},[476],[63,13524,13526,13546],{"className":13525},[411,412],[63,13527,13529,13543],{"className":13528},[416],[63,13530,13532],{"className":13531,"style":3157},[420],[63,13533,13534,13537],{"style":1535},[63,13535],{"className":13536,"style":817},[428],[63,13538,13540],{"className":13539},[433,434,435,436],[63,13541,363],{"className":13542},[399,436],[63,13544,507],{"className":13545},[506],[63,13547,13549],{"className":13548},[416],[63,13550,13552],{"className":13551,"style":1554},[420],[63,13553],{},[63,13555],{"className":13556,"style":1177},[776],[63,13558,3036],{"className":13559},[1222],[63,13561],{"className":13562,"style":1177},[776],[63,13564,13566,13569],{"className":13565},[399],[63,13567,12890],{"className":13568},[399,472],[63,13570,13572],{"className":13571},[476],[63,13573,13575,13596],{"className":13574},[411,412],[63,13576,13578,13593],{"className":13577},[416],[63,13579,13582],{"className":13580,"style":13581},[420],"height:0.3283em;",[63,13583,13584,13587],{"style":1535},[63,13585],{"className":13586,"style":817},[428],[63,13588,13590],{"className":13589},[433,434,435,436],[63,13591,13358],{"className":13592,"style":3251},[399,472,436],[63,13594,507],{"className":13595},[506],[63,13597,13599],{"className":13598},[416],[63,13600,13602],{"className":13601,"style":1554},[420],[63,13603],{},[63,13605,1092],{"className":13606},[593],[14,13608,13609,13610,13829,13830,14070],{},"环境的状态转移满足 ",[63,13611,13613,13659],{"className":13612},[342],[63,13614,13616],{"className":13615},[346],[348,13617,13618],{"xmlns":350},[352,13619,13620,13656],{},[355,13621,13622,13634,13636,13638,13640,13646,13648,13654],{},[368,13623,13624,13626],{},[371,13625,12890],{},[355,13627,13628,13630,13632],{},[371,13629,1057],{},[738,13631,5839],{},[361,13633,363],{},[738,13635,204],{},[371,13637,11741],{},[738,13639,1078],{"stretchy":1077},[368,13641,13642,13644],{},[371,13643,12890],{},[371,13645,1057],{},[738,13647,1083],{"separator":386},[368,13649,13650,13652],{},[371,13651,38],{},[371,13653,1057],{},[738,13655,1092],{"stretchy":1077},[378,13657,13658],{"encoding":380},"s_{t+1} = f(s_t,a_t)",[63,13660,13662,13727],{"className":13661,"ariaHidden":386},[385],[63,13663,13665,13669,13718,13721,13724],{"className":13664},[390],[63,13666],{"className":13667,"style":13668},[394],"height:0.6389em;vertical-align:-0.2083em;",[63,13670,13672,13675],{"className":13671},[399],[63,13673,12890],{"className":13674},[399,472],[63,13676,13678],{"className":13677},[476],[63,13679,13681,13710],{"className":13680},[411,412],[63,13682,13684,13707],{"className":13683},[416],[63,13685,13687],{"className":13686,"style":3157},[420],[63,13688,13689,13692],{"style":1535},[63,13690],{"className":13691,"style":817},[428],[63,13693,13695],{"className":13694},[433,434,435,436],[63,13696,13698,13701,13704],{"className":13697},[399,436],[63,13699,1057],{"className":13700},[399,472,436],[63,13702,5839],{"className":13703},[830,436],[63,13705,363],{"className":13706},[399,436],[63,13708,507],{"className":13709},[506],[63,13711,13713],{"className":13712},[416],[63,13714,13716],{"className":13715,"style":6959},[420],[63,13717],{},[63,13719],{"className":13720,"style":777},[776],[63,13722,204],{"className":13723},[781],[63,13725],{"className":13726,"style":777},[776],[63,13728,13730,13733,13737,13740,13780,13783,13786,13826],{"className":13729},[390],[63,13731],{"className":13732,"style":1155},[394],[63,13734,11741],{"className":13735,"style":13736},[399,472],"margin-right:0.10764em;",[63,13738,1078],{"className":13739},[403],[63,13741,13743,13746],{"className":13742},[399],[63,13744,12890],{"className":13745},[399,472],[63,13747,13749],{"className":13748},[476],[63,13750,13752,13772],{"className":13751},[411,412],[63,13753,13755,13769],{"className":13754},[416],[63,13756,13758],{"className":13757,"style":12818},[420],[63,13759,13760,13763],{"style":1535},[63,13761],{"className":13762,"style":817},[428],[63,13764,13766],{"className":13765},[433,434,435,436],[63,13767,1057],{"className":13768},[399,472,436],[63,13770,507],{"className":13771},[506],[63,13773,13775],{"className":13774},[416],[63,13776,13778],{"className":13777,"style":1554},[420],[63,13779],{},[63,13781,1083],{"className":13782},[1173],[63,13784],{"className":13785,"style":1177},[776],[63,13787,13789,13792],{"className":13788},[399],[63,13790,38],{"className":13791},[399,472],[63,13793,13795],{"className":13794},[476],[63,13796,13798,13818],{"className":13797},[411,412],[63,13799,13801,13815],{"className":13800},[416],[63,13802,13804],{"className":13803,"style":12818},[420],[63,13805,13806,13809],{"style":1535},[63,13807],{"className":13808,"style":817},[428],[63,13810,13812],{"className":13811},[433,434,435,436],[63,13813,1057],{"className":13814},[399,472,436],[63,13816,507],{"className":13817},[506],[63,13819,13821],{"className":13820},[416],[63,13822,13824],{"className":13823,"style":1554},[420],[63,13825],{},[63,13827,1092],{"className":13828},[593],"（确定性环境）或 ",[63,13831,13833,13884],{"className":13832},[342],[63,13834,13836],{"className":13835},[346],[348,13837,13838],{"xmlns":350},[352,13839,13840,13881],{},[355,13841,13842,13854,13857,13859,13861,13863,13865,13871,13873,13879],{},[368,13843,13844,13846],{},[371,13845,12890],{},[355,13847,13848,13850,13852],{},[371,13849,1057],{},[738,13851,5839],{},[361,13853,363],{},[738,13855,13856],{},"∼",[371,13858,5685],{},[738,13860,1078],{"stretchy":1077},[738,13862,1864],{},[738,13864,13030],{},[368,13866,13867,13869],{},[371,13868,12890],{},[371,13870,1057],{},[738,13872,1083],{"separator":386},[368,13874,13875,13877],{},[371,13876,38],{},[371,13878,1057],{},[738,13880,1092],{"stretchy":1077},[378,13882,13883],{"encoding":380},"s_{t+1} \\sim P(\\cdot \\mid s_t,a_t)",[63,13885,13887,13951,13975],{"className":13886,"ariaHidden":386},[385],[63,13888,13890,13893,13942,13945,13948],{"className":13889},[390],[63,13891],{"className":13892,"style":13668},[394],[63,13894,13896,13899],{"className":13895},[399],[63,13897,12890],{"className":13898},[399,472],[63,13900,13902],{"className":13901},[476],[63,13903,13905,13934],{"className":13904},[411,412],[63,13906,13908,13931],{"className":13907},[416],[63,13909,13911],{"className":13910,"style":3157},[420],[63,13912,13913,13916],{"style":1535},[63,13914],{"className":13915,"style":817},[428],[63,13917,13919],{"className":13918},[433,434,435,436],[63,13920,13922,13925,13928],{"className":13921},[399,436],[63,13923,1057],{"className":13924},[399,472,436],[63,13926,5839],{"className":13927},[830,436],[63,13929,363],{"className":13930},[399,436],[63,13932,507],{"className":13933},[506],[63,13935,13937],{"className":13936},[416],[63,13938,13940],{"className":13939,"style":6959},[420],[63,13941],{},[63,13943],{"className":13944,"style":777},[776],[63,13946,13856],{"className":13947},[781],[63,13949],{"className":13950,"style":777},[776],[63,13952,13954,13957,13960,13963,13966,13969,13972],{"className":13953},[390],[63,13955],{"className":13956,"style":1155},[394],[63,13958,5685],{"className":13959,"style":3251},[399,472],[63,13961,1078],{"className":13962},[403],[63,13964,1864],{"className":13965},[399],[63,13967],{"className":13968,"style":777},[776],[63,13970,13030],{"className":13971},[781],[63,13973],{"className":13974,"style":777},[776],[63,13976,13978,13981,14021,14024,14027,14067],{"className":13977},[390],[63,13979],{"className":13980,"style":1155},[394],[63,13982,13984,13987],{"className":13983},[399],[63,13985,12890],{"className":13986},[399,472],[63,13988,13990],{"className":13989},[476],[63,13991,13993,14013],{"className":13992},[411,412],[63,13994,13996,14010],{"className":13995},[416],[63,13997,13999],{"className":13998,"style":12818},[420],[63,14000,14001,14004],{"style":1535},[63,14002],{"className":14003,"style":817},[428],[63,14005,14007],{"className":14006},[433,434,435,436],[63,14008,1057],{"className":14009},[399,472,436],[63,14011,507],{"className":14012},[506],[63,14014,14016],{"className":14015},[416],[63,14017,14019],{"className":14018,"style":1554},[420],[63,14020],{},[63,14022,1083],{"className":14023},[1173],[63,14025],{"className":14026,"style":1177},[776],[63,14028,14030,14033],{"className":14029},[399],[63,14031,38],{"className":14032},[399,472],[63,14034,14036],{"className":14035},[476],[63,14037,14039,14059],{"className":14038},[411,412],[63,14040,14042,14056],{"className":14041},[416],[63,14043,14045],{"className":14044,"style":12818},[420],[63,14046,14047,14050],{"style":1535},[63,14048],{"className":14049,"style":817},[428],[63,14051,14053],{"className":14052},[433,434,435,436],[63,14054,1057],{"className":14055},[399,472,436],[63,14057,507],{"className":14058},[506],[63,14060,14062],{"className":14061},[416],[63,14063,14065],{"className":14064,"style":1554},[420],[63,14066],{},[63,14068,1092],{"className":14069},[593],"（随机性环境）。",[14,14072,14073,14076],{},[11217,14074,14075],{},"Return","（回报）指从当前时间步到 episode 结束所获得奖励的累积和（或带折扣的累积和）。",[10,14078,14079],{"id":14079},"训练目标",[14,14081,14082,14083,14154],{},"我们要训练一个策略网络 ",[63,14084,14086,14105],{"className":14085},[342],[63,14087,14089],{"className":14088},[346],[348,14090,14091],{"xmlns":350},[352,14092,14093,14102],{},[355,14094,14095],{},[368,14096,14097,14099],{},[371,14098,6764],{},[371,14100,14101],{},"θ",[378,14103,14104],{"encoding":380},"\\pi_\\theta",[63,14106,14108],{"className":14107,"ariaHidden":386},[385],[63,14109,14111,14114],{"className":14110},[390],[63,14112],{"className":14113,"style":12799},[394],[63,14115,14117,14120],{"className":14116},[399],[63,14118,6764],{"className":14119,"style":718},[399,472],[63,14121,14123],{"className":14122},[476],[63,14124,14126,14146],{"className":14125},[411,412],[63,14127,14129,14143],{"className":14128},[416],[63,14130,14132],{"className":14131,"style":1532},[420],[63,14133,14134,14137],{"style":2083},[63,14135],{"className":14136,"style":817},[428],[63,14138,14140],{"className":14139},[433,434,435,436],[63,14141,14101],{"className":14142,"style":3276},[399,472,436],[63,14144,507],{"className":14145},[506],[63,14147,14149],{"className":14148},[416],[63,14150,14152],{"className":14151,"style":1554},[420],[63,14153],{},"，使其在所有可能的状态下做出动作后，期望回报最大化：",[14,14156,14157],{},[63,14158,14160,14235],{"className":14159},[342],[63,14161,14163],{"className":14162},[346],[348,14164,14165],{"xmlns":350},[352,14166,14167,14232],{},[355,14168,14169,14171,14173,14175,14177,14179,14181,14203,14205,14212,14214,14216,14218,14220,14226,14228,14230],{},[371,14170,5688],{},[738,14172,1078],{"stretchy":1077},[371,14174,747],{},[738,14176,1078],{"stretchy":1077},[371,14178,13274],{},[738,14180,1092],{"stretchy":1077},[368,14182,14183,14185],{},[738,14184,1092],{"stretchy":1077},[355,14186,14187,14189,14191,14197,14199,14201],{},[371,14188,13274],{},[738,14190,13856],{},[368,14192,14193,14195],{},[371,14194,5685],{},[371,14196,14101],{},[738,14198,1078],{"stretchy":1077},[371,14200,13274],{},[738,14202,1092],{"stretchy":1077},[738,14204,204],{},[14206,14207,14208,14210],"munder",{},[738,14209,1874],{},[371,14211,13274],{},[371,14213,747],{},[738,14215,1078],{"stretchy":1077},[371,14217,13274],{},[738,14219,1092],{"stretchy":1077},[368,14221,14222,14224],{},[371,14223,5685],{},[371,14225,14101],{},[738,14227,1078],{"stretchy":1077},[371,14229,13274],{},[738,14231,1092],{"stretchy":1077},[378,14233,14234],{"encoding":380}," E (R(\\tau))_{\\tau \\sim P_{\\theta}(\\tau)} = \\sum\\limits_\\tau R(\\tau)P_\\theta(\\tau)",[63,14236,14238,14373],{"className":14237,"ariaHidden":386},[385],[63,14239,14241,14245,14248,14251,14255,14258,14261,14264,14364,14367,14370],{"className":14240},[390],[63,14242],{"className":14243,"style":14244},[394],"height:1.1052em;vertical-align:-0.3552em;",[63,14246,5688],{"className":14247,"style":6101},[399,472],[63,14249,1078],{"className":14250},[403],[63,14252,747],{"className":14253,"style":14254},[399,472],"margin-right:0.00773em;",[63,14256,1078],{"className":14257},[403],[63,14259,13274],{"className":14260,"style":13290},[399,472],[63,14262,1092],{"className":14263},[593],[63,14265,14267,14270],{"className":14266},[593],[63,14268,1092],{"className":14269},[593],[63,14271,14273],{"className":14272},[476],[63,14274,14276,14356],{"className":14275},[411,412],[63,14277,14279,14353],{"className":14278},[416],[63,14280,14282],{"className":14281,"style":486},[420],[63,14283,14285,14288],{"style":14284},"top:-2.5198em;margin-left:0em;margin-right:0.05em;",[63,14286],{"className":14287,"style":817},[428],[63,14289,14291],{"className":14290},[433,434,435,436],[63,14292,14294,14297,14300,14344,14347,14350],{"className":14293},[399,436],[63,14295,13274],{"className":14296,"style":13290},[399,472,436],[63,14298,13856],{"className":14299},[781,436],[63,14301,14303,14306],{"className":14302},[399,436],[63,14304,5685],{"className":14305,"style":3251},[399,472,436],[63,14307,14309],{"className":14308},[476],[63,14310,14312,14336],{"className":14311},[411,412],[63,14313,14315,14333],{"className":14314},[416],[63,14316,14318],{"className":14317,"style":486},[420],[63,14319,14321,14324],{"style":14320},"top:-2.3488em;margin-left:-0.1389em;margin-right:0.0714em;",[63,14322],{"className":14323,"style":493},[428],[63,14325,14327],{"className":14326},[433,497,498,436],[63,14328,14330],{"className":14329},[399,436],[63,14331,14101],{"className":14332,"style":3276},[399,472,436],[63,14334,507],{"className":14335},[506],[63,14337,14339],{"className":14338},[416],[63,14340,14342],{"className":14341,"style":514},[420],[63,14343],{},[63,14345,1078],{"className":14346},[403,436],[63,14348,13274],{"className":14349,"style":13290},[399,472,436],[63,14351,1092],{"className":14352},[593,436],[63,14354,507],{"className":14355},[506],[63,14357,14359],{"className":14358},[416],[63,14360,14362],{"className":14361,"style":6153},[420],[63,14363],{},[63,14365],{"className":14366,"style":777},[776],[63,14368,204],{"className":14369},[781],[63,14371],{"className":14372,"style":777},[776],[63,14374,14376,14380,14428,14431,14434,14437,14440,14443,14483,14486,14489],{"className":14375},[390],[63,14377],{"className":14378,"style":14379},[394],"height:1.7em;vertical-align:-0.95em;",[63,14381,14384],{"className":14382},[1212,14383],"op-limits",[63,14385,14387,14419],{"className":14386},[411,412],[63,14388,14390,14416],{"className":14389},[416],[63,14391,14394,14406],{"className":14392,"style":14393},[420],"height:0.75em;",[63,14395,14397,14400],{"style":14396},"top:-2.15em;margin-left:0em;",[63,14398],{"className":14399,"style":429},[428],[63,14401,14403],{"className":14402},[433,434,435,436],[63,14404,13274],{"className":14405,"style":13290},[399,472,436],[63,14407,14408,14411],{"style":458},[63,14409],{"className":14410,"style":429},[428],[63,14412,14413],{},[63,14414,1874],{"className":14415},[1212,1958,1959],[63,14417,507],{"className":14418},[506],[63,14420,14422],{"className":14421},[416],[63,14423,14426],{"className":14424,"style":14425},[420],"height:0.95em;",[63,14427],{},[63,14429],{"className":14430,"style":1177},[776],[63,14432,747],{"className":14433,"style":14254},[399,472],[63,14435,1078],{"className":14436},[403],[63,14438,13274],{"className":14439,"style":13290},[399,472],[63,14441,1092],{"className":14442},[593],[63,14444,14446,14449],{"className":14445},[399],[63,14447,5685],{"className":14448,"style":3251},[399,472],[63,14450,14452],{"className":14451},[476],[63,14453,14455,14475],{"className":14454},[411,412],[63,14456,14458,14472],{"className":14457},[416],[63,14459,14461],{"className":14460,"style":1532},[420],[63,14462,14463,14466],{"style":7483},[63,14464],{"className":14465,"style":817},[428],[63,14467,14469],{"className":14468},[433,434,435,436],[63,14470,14101],{"className":14471,"style":3276},[399,472,436],[63,14473,507],{"className":14474},[506],[63,14476,14478],{"className":14477},[416],[63,14479,14481],{"className":14480,"style":1554},[420],[63,14482],{},[63,14484,1078],{"className":14485},[403],[63,14487,13274],{"className":14488,"style":13290},[399,472],[63,14490,1092],{"className":14491},[593],[63,14493,14496],{"className":14494},[14495],"katex-display",[63,14497,14499,15279],{"className":14498},[342],[63,14500,14502],{"className":14501},[346],[348,14503,14504],{"xmlns":350,"display":9561},[352,14505,14506,15276],{},[14507,14508,14512,14598,14644,14724,14802,14896,14969,15072,15174],"mtable",{"rowspacing":14509,"columnalign":14510,"columnspacing":14511},"0.25em","right left","0em",[14513,14514,14515,14560],"mtr",{},[14516,14517,14518],"mtd",{},[14519,14520,14521],"mstyle",{"scriptlevel":1802,"displaystyle":386},[355,14522,14523,14526,14528,14530,14532,14534,14536,14538],{},[371,14524,14525],{"mathvariant":1053},"∇",[371,14527,5688],{},[738,14529,1078],{"stretchy":1077},[371,14531,747],{},[738,14533,1078],{"stretchy":1077},[371,14535,13274],{},[738,14537,1092],{"stretchy":1077},[368,14539,14540,14542],{},[738,14541,1092],{"stretchy":1077},[355,14543,14544,14546,14548,14554,14556,14558],{},[371,14545,13274],{},[738,14547,13856],{},[368,14549,14550,14552],{},[371,14551,5685],{},[371,14553,14101],{},[738,14555,1078],{"stretchy":1077},[371,14557,13274],{},[738,14559,1092],{"stretchy":1077},[14516,14561,14562],{},[14519,14563,14564],{"scriptlevel":1802,"displaystyle":386},[355,14565,14566,14568,14570,14572,14578,14580,14582,14584,14586,14592,14594,14596],{},[355,14567],{},[738,14569,204],{},[371,14571,14525],{"mathvariant":1053},[14206,14573,14574,14576],{},[738,14575,1874],{},[371,14577,13274],{},[371,14579,747],{},[738,14581,1078],{"stretchy":1077},[371,14583,13274],{},[738,14585,1092],{"stretchy":1077},[368,14587,14588,14590],{},[371,14589,5685],{},[371,14591,14101],{},[738,14593,1078],{"stretchy":1077},[371,14595,13274],{},[738,14597,1092],{"stretchy":1077},[14513,14599,14600,14606],{},[14516,14601,14602],{},[14519,14603,14604],{"scriptlevel":1802,"displaystyle":386},[355,14605],{},[14516,14607,14608],{},[14519,14609,14610],{"scriptlevel":1802,"displaystyle":386},[355,14611,14612,14614,14616,14622,14624,14626,14628,14630,14632,14638,14640,14642],{},[355,14613],{},[738,14615,204],{},[14206,14617,14618,14620],{},[738,14619,1874],{},[371,14621,13274],{},[371,14623,747],{},[738,14625,1078],{"stretchy":1077},[371,14627,13274],{},[738,14629,1092],{"stretchy":1077},[371,14631,14525],{"mathvariant":1053},[368,14633,14634,14636],{},[371,14635,5685],{},[371,14637,14101],{},[738,14639,1078],{"stretchy":1077},[371,14641,13274],{},[738,14643,1092],{"stretchy":1077},[14513,14645,14646,14652],{},[14516,14647,14648],{},[14519,14649,14650],{"scriptlevel":1802,"displaystyle":386},[355,14651],{},[14516,14653,14654],{},[14519,14655,14656],{"scriptlevel":1802,"displaystyle":386},[355,14657,14658,14660,14662,14668,14670,14672,14674,14676,14678,14684,14686,14688,14690,14692],{},[355,14659],{},[738,14661,204],{},[14206,14663,14664,14666],{},[738,14665,1874],{},[371,14667,13274],{},[371,14669,747],{},[738,14671,1078],{"stretchy":1077},[371,14673,13274],{},[738,14675,1092],{"stretchy":1077},[371,14677,14525],{"mathvariant":1053},[368,14679,14680,14682],{},[371,14681,5685],{},[371,14683,14101],{},[738,14685,1078],{"stretchy":1077},[371,14687,13274],{},[738,14689,1092],{"stretchy":1077},[738,14691,1864],{},[14519,14693,14694],{"scriptlevel":1802,"displaystyle":386},[358,14695,14696,14710],{},[355,14697,14698,14704,14706,14708],{},[368,14699,14700,14702],{},[371,14701,5685],{},[371,14703,14101],{},[738,14705,1078],{"stretchy":1077},[371,14707,13274],{},[738,14709,1092],{"stretchy":1077},[355,14711,14712,14718,14720,14722],{},[368,14713,14714,14716],{},[371,14715,5685],{},[371,14717,14101],{},[738,14719,1078],{"stretchy":1077},[371,14721,13274],{},[738,14723,1092],{"stretchy":1077},[14513,14725,14726,14732],{},[14516,14727,14728],{},[14519,14729,14730],{"scriptlevel":1802,"displaystyle":386},[355,14731],{},[14516,14733,14734],{},[14519,14735,14736],{"scriptlevel":1802,"displaystyle":386},[355,14737,14738,14740,14742,14748,14754,14756,14758,14760,14762,14764,14766,14768],{},[355,14739],{},[738,14741,204],{},[14206,14743,14744,14746],{},[738,14745,1874],{},[371,14747,13274],{},[368,14749,14750,14752],{},[371,14751,5685],{},[371,14753,14101],{},[738,14755,1078],{"stretchy":1077},[371,14757,13274],{},[738,14759,1092],{"stretchy":1077},[371,14761,747],{},[738,14763,1078],{"stretchy":1077},[371,14765,13274],{},[738,14767,1092],{"stretchy":1077},[14519,14769,14770],{"scriptlevel":1802,"displaystyle":386},[358,14771,14772,14788],{},[355,14773,14774,14776,14782,14784,14786],{},[371,14775,14525],{"mathvariant":1053},[368,14777,14778,14780],{},[371,14779,5685],{},[371,14781,14101],{},[738,14783,1078],{"stretchy":1077},[371,14785,13274],{},[738,14787,1092],{"stretchy":1077},[355,14789,14790,14796,14798,14800],{},[368,14791,14792,14794],{},[371,14793,5685],{},[371,14795,14101],{},[738,14797,1078],{"stretchy":1077},[371,14799,13274],{},[738,14801,1092],{"stretchy":1077},[14513,14803,14804,14810],{},[14516,14805,14806],{},[14519,14807,14808],{"scriptlevel":1802,"displaystyle":386},[355,14809],{},[14516,14811,14812],{},[14519,14813,14814],{"scriptlevel":1802,"displaystyle":386},[355,14815,14816,14818,14821,14827,14842,14844,14846,14852,14854],{},[355,14817],{},[738,14819,14820],{},"≈",[358,14822,14823,14825],{},[361,14824,363],{},[371,14826,7342],{},[14828,14829,14830,14832,14840],"munderover",{},[738,14831,1874],{},[355,14833,14834,14836,14838],{},[371,14835,612],{},[738,14837,204],{},[361,14839,363],{},[371,14841,7342],{},[371,14843,747],{},[738,14845,1078],{"stretchy":1077},[742,14847,14848,14850],{},[371,14849,13274],{},[371,14851,612],{},[738,14853,1092],{"stretchy":1077},[14519,14855,14856],{"scriptlevel":1802,"displaystyle":386},[358,14857,14858,14878],{},[355,14859,14860,14862,14868,14870,14876],{},[371,14861,14525],{"mathvariant":1053},[368,14863,14864,14866],{},[371,14865,14],{},[371,14867,14101],{},[738,14869,1078],{"stretchy":1077},[742,14871,14872,14874],{},[371,14873,13274],{},[371,14875,612],{},[738,14877,1092],{"stretchy":1077},[355,14879,14880,14886,14888,14894],{},[368,14881,14882,14884],{},[371,14883,5685],{},[371,14885,14101],{},[738,14887,1078],{"stretchy":1077},[742,14889,14890,14892],{},[371,14891,13274],{},[371,14893,612],{},[738,14895,1092],{"stretchy":1077},[14513,14897,14898,14904],{},[14516,14899,14900],{},[14519,14901,14902],{"scriptlevel":1802,"displaystyle":386},[355,14903],{},[14516,14905,14906],{},[14519,14907,14908],{"scriptlevel":1802,"displaystyle":386},[355,14909,14910,14912,14914,14920,14934,14936,14938,14944,14946,14948,14951,14953,14959,14961,14967],{},[355,14911],{},[738,14913,204],{},[358,14915,14916,14918],{},[361,14917,363],{},[371,14919,7342],{},[14828,14921,14922,14924,14932],{},[738,14923,1874],{},[355,14925,14926,14928,14930],{},[371,14927,612],{},[738,14929,204],{},[361,14931,363],{},[371,14933,7342],{},[371,14935,747],{},[738,14937,1078],{"stretchy":1077},[742,14939,14940,14942],{},[371,14941,13274],{},[371,14943,612],{},[738,14945,1092],{"stretchy":1077},[371,14947,14525],{"mathvariant":1053},[371,14949,14950],{},"log",[738,14952,1100],{},[368,14954,14955,14957],{},[371,14956,5685],{},[371,14958,14101],{},[738,14960,1078],{"stretchy":1077},[742,14962,14963,14965],{},[371,14964,13274],{},[371,14966,612],{},[738,14968,1092],{"stretchy":1077},[14513,14970,14971,14977],{},[14516,14972,14973],{},[14519,14974,14975],{"scriptlevel":1802,"displaystyle":386},[355,14976],{},[14516,14978,14979],{},[14519,14980,14981],{"scriptlevel":1802,"displaystyle":386},[355,14982,14983,14985,14987,14993,15007,15009,15011,15017,15019,15021,15023,15025,15044,15050,15052,15060,15062,15070],{},[355,14984],{},[738,14986,204],{},[358,14988,14989,14991],{},[361,14990,363],{},[371,14992,7342],{},[14828,14994,14995,14997,15005],{},[738,14996,1874],{},[355,14998,14999,15001,15003],{},[371,15000,612],{},[738,15002,204],{},[361,15004,363],{},[371,15006,7342],{},[371,15008,747],{},[738,15010,1078],{"stretchy":1077},[742,15012,15013,15015],{},[371,15014,13274],{},[371,15016,612],{},[738,15018,1092],{"stretchy":1077},[371,15020,14525],{"mathvariant":1053},[371,15022,14950],{},[738,15024,1100],{},[14828,15026,15027,15030,15038],{},[738,15028,15029],{},"∏",[355,15031,15032,15034,15036],{},[371,15033,1057],{},[738,15035,204],{},[361,15037,363],{},[368,15039,15040,15042],{},[371,15041,13358],{},[371,15043,612],{},[368,15045,15046,15048],{},[371,15047,5685],{},[371,15049,14101],{},[738,15051,1078],{"stretchy":1077},[1870,15053,15054,15056,15058],{},[371,15055,38],{},[371,15057,612],{},[371,15059,1057],{},[371,15061,13030],{"mathvariant":1053},[1870,15063,15064,15066,15068],{},[371,15065,12890],{},[371,15067,612],{},[371,15069,1057],{},[738,15071,1092],{"stretchy":1077},[14513,15073,15074,15080],{},[14516,15075,15076],{},[14519,15077,15078],{"scriptlevel":1802,"displaystyle":386},[355,15079],{},[14516,15081,15082],{},[14519,15083,15084],{"scriptlevel":1802,"displaystyle":386},[355,15085,15086,15088,15090,15096,15110,15112,15114,15120,15122,15140,15142,15144,15146,15152,15154,15162,15164,15172],{},[355,15087],{},[738,15089,204],{},[358,15091,15092,15094],{},[361,15093,363],{},[371,15095,7342],{},[14828,15097,15098,15100,15108],{},[738,15099,1874],{},[355,15101,15102,15104,15106],{},[371,15103,612],{},[738,15105,204],{},[361,15107,363],{},[371,15109,7342],{},[371,15111,747],{},[738,15113,1078],{"stretchy":1077},[742,15115,15116,15118],{},[371,15117,13274],{},[371,15119,612],{},[738,15121,1092],{"stretchy":1077},[14828,15123,15124,15126,15134],{},[738,15125,1874],{},[355,15127,15128,15130,15132],{},[371,15129,1057],{},[738,15131,204],{},[361,15133,363],{},[368,15135,15136,15138],{},[371,15137,13358],{},[371,15139,612],{},[371,15141,14525],{"mathvariant":1053},[371,15143,14950],{},[738,15145,1100],{},[368,15147,15148,15150],{},[371,15149,5685],{},[371,15151,14101],{},[738,15153,1078],{"stretchy":1077},[1870,15155,15156,15158,15160],{},[371,15157,38],{},[371,15159,612],{},[371,15161,1057],{},[371,15163,13030],{"mathvariant":1053},[1870,15165,15166,15168,15170],{},[371,15167,12890],{},[371,15169,612],{},[371,15171,1057],{},[738,15173,1092],{"stretchy":1077},[14513,15175,15176,15182],{},[14516,15177,15178],{},[14519,15179,15180],{"scriptlevel":1802,"displaystyle":386},[355,15181],{},[14516,15183,15184],{},[14519,15185,15186],{"scriptlevel":1802,"displaystyle":386},[355,15187,15188,15190,15192,15198,15212,15230,15232,15234,15240,15242,15244,15246,15248,15254,15256,15264,15266,15274],{},[355,15189],{},[738,15191,204],{},[358,15193,15194,15196],{},[361,15195,363],{},[371,15197,7342],{},[14828,15199,15200,15202,15210],{},[738,15201,1874],{},[355,15203,15204,15206,15208],{},[371,15205,612],{},[738,15207,204],{},[361,15209,363],{},[371,15211,7342],{},[14828,15213,15214,15216,15224],{},[738,15215,1874],{},[355,15217,15218,15220,15222],{},[371,15219,1057],{},[738,15221,204],{},[361,15223,363],{},[368,15225,15226,15228],{},[371,15227,13358],{},[371,15229,612],{},[371,15231,747],{},[738,15233,1078],{"stretchy":1077},[742,15235,15236,15238],{},[371,15237,13274],{},[371,15239,612],{},[738,15241,1092],{"stretchy":1077},[371,15243,14525],{"mathvariant":1053},[371,15245,14950],{},[738,15247,1100],{},[368,15249,15250,15252],{},[371,15251,5685],{},[371,15253,14101],{},[738,15255,1078],{"stretchy":1077},[1870,15257,15258,15260,15262],{},[371,15259,38],{},[371,15261,612],{},[371,15263,1057],{},[371,15265,13030],{"mathvariant":1053},[1870,15267,15268,15270,15272],{},[371,15269,12890],{},[371,15271,612],{},[371,15273,1057],{},[738,15275,1092],{"stretchy":1077},[378,15277,15278],{"encoding":380},"\\begin{aligned}\n\\nabla E(R(\\tau))_{\\tau \\sim P_\\theta(\\tau)} &= \\nabla \\sum_\\tau R(\\tau)P_\\theta(\\tau)\\\\\n&= \\sum_\\tau R(\\tau)\\nabla P_\\theta(\\tau)\\\\\n&= \\sum_\\tau R(\\tau)\\nabla P_\\theta(\\tau) \\cdot \\dfrac{P_\\theta(\\tau)}{P_\\theta(\\tau)}\\\\\n&= \\sum_\\tau P_\\theta(\\tau)R(\\tau) \\dfrac{\\nabla P_\\theta(\\tau)}{P_\\theta(\\tau)}\\\\\n&\\approx \\frac{1}{N}\\sum\\limits_{n=1}^N R(\\tau^n)\\dfrac{\\nabla p_\\theta(\\tau^n)}{P_\\theta(\\tau^n)}\\\\\n&= \\frac{1}{N}\\sum\\limits_{n=1}^N R(\\tau^n)\\nabla \\log P_\\theta(\\tau^n)\\\\\n&= \\frac{1}{N}\\sum\\limits_{n=1}^N R(\\tau^n)\\nabla \\log \\prod\\limits_{t=1}^{T_n}P_\\theta(a_n^t|s_n^t)\\\\\n&= \\frac{1}{N}\\sum\\limits_{n=1}^N R(\\tau^n)\\sum\\limits_{t=1}^{T_n}\\nabla \\log P_\\theta(a_n^t|s_n^t)\\\\\n&= \\frac{1}{N}\\sum\\limits_{n=1}^N \\sum\\limits_{t=1}^{T_n} R(\\tau^n)\\nabla \\log P_\\theta(a_n^t|s_n^t)\\\\\n\\end{aligned}",[63,15280,15282],{"className":15281,"ariaHidden":386},[385],[63,15283,15285,15289],{"className":15284},[390],[63,15286],{"className":15287,"style":15288},[394],"height:28.1646em;vertical-align:-13.8323em;",[63,15290,15292],{"className":15291},[399],[63,15293,15295,15519],{"className":15294},[14507],[63,15296,15299],{"className":15297},[15298],"col-align-r",[63,15300,15302,15510],{"className":15301},[411,412],[63,15303,15305,15507],{"className":15304},[416],[63,15306,15309,15435,15444,15453,15462,15471,15480,15489,15498],{"className":15307,"style":15308},[420],"height:14.3323em;",[63,15310,15312,15316],{"style":15311},"top:-17.1217em;",[63,15313],{"className":15314,"style":15315},[428],"height:3.8394em;",[63,15317,15319,15322,15325,15328,15331,15334,15337,15340],{"className":15318},[399],[63,15320,14525],{"className":15321},[399],[63,15323,5688],{"className":15324,"style":6101},[399,472],[63,15326,1078],{"className":15327},[403],[63,15329,747],{"className":15330,"style":14254},[399,472],[63,15332,1078],{"className":15333},[403],[63,15335,13274],{"className":15336,"style":13290},[399,472],[63,15338,1092],{"className":15339},[593],[63,15341,15343,15346],{"className":15342},[593],[63,15344,1092],{"className":15345},[593],[63,15347,15349],{"className":15348},[476],[63,15350,15352,15427],{"className":15351},[411,412],[63,15353,15355,15424],{"className":15354},[416],[63,15356,15358],{"className":15357,"style":486},[420],[63,15359,15360,15363],{"style":14284},[63,15361],{"className":15362,"style":817},[428],[63,15364,15366],{"className":15365},[433,434,435,436],[63,15367,15369,15372,15375,15415,15418,15421],{"className":15368},[399,436],[63,15370,13274],{"className":15371,"style":13290},[399,472,436],[63,15373,13856],{"className":15374},[781,436],[63,15376,15378,15381],{"className":15377},[399,436],[63,15379,5685],{"className":15380,"style":3251},[399,472,436],[63,15382,15384],{"className":15383},[476],[63,15385,15387,15407],{"className":15386},[411,412],[63,15388,15390,15404],{"className":15389},[416],[63,15391,15393],{"className":15392,"style":486},[420],[63,15394,15395,15398],{"style":14320},[63,15396],{"className":15397,"style":493},[428],[63,15399,15401],{"className":15400},[433,497,498,436],[63,15402,14101],{"className":15403,"style":3276},[399,472,436],[63,15405,507],{"className":15406},[506],[63,15408,15410],{"className":15409},[416],[63,15411,15413],{"className":15412,"style":514},[420],[63,15414],{},[63,15416,1078],{"className":15417},[403,436],[63,15419,13274],{"className":15420,"style":13290},[399,472,436],[63,15422,1092],{"className":15423},[593,436],[63,15425,507],{"className":15426},[506],[63,15428,15430],{"className":15429},[416],[63,15431,15433],{"className":15432,"style":6153},[420],[63,15434],{},[63,15436,15438,15441],{"style":15437},"top:-14.5217em;",[63,15439],{"className":15440,"style":15315},[428],[63,15442],{"className":15443},[399],[63,15445,15447,15450],{"style":15446},"top:-11.5447em;",[63,15448],{"className":15449,"style":15315},[428],[63,15451],{"className":15452},[399],[63,15454,15456,15459],{"style":15455},"top:-8.5677em;",[63,15457],{"className":15458,"style":15315},[428],[63,15460],{"className":15461},[399],[63,15463,15465,15468],{"style":15464},"top:-5.1894em;",[63,15466],{"className":15467,"style":15315},[428],[63,15469],{"className":15470},[399],[63,15472,15474,15477],{"style":15473},"top:-1.7939em;",[63,15475],{"className":15476,"style":15315},[428],[63,15478],{"className":15479},[399],[63,15481,15483,15486],{"style":15482},"top:1.6126em;",[63,15484],{"className":15485,"style":15315},[428],[63,15487],{"className":15488},[399],[63,15490,15492,15495],{"style":15491},"top:5.0192em;",[63,15493],{"className":15494,"style":15315},[428],[63,15496],{"className":15497},[399],[63,15499,15501,15504],{"style":15500},"top:8.4257em;",[63,15502],{"className":15503,"style":15315},[428],[63,15505],{"className":15506},[399],[63,15508,507],{"className":15509},[506],[63,15511,15513],{"className":15512},[416],[63,15514,15517],{"className":15515,"style":15516},[420],"height:13.8323em;",[63,15518],{},[63,15520,15523],{"className":15521},[15522],"col-align-l",[63,15524,15526,18477],{"className":15525},[411,412],[63,15527,15529,18474],{"className":15528},[416],[63,15530,15532,15672,15803,16101,16386,16797,17079,17547,18012],{"className":15531,"style":15308},[420],[63,15533,15534,15537],{"style":15311},[63,15535],{"className":15536,"style":15315},[428],[63,15538,15540,15543,15546,15549,15552,15555,15558,15608,15611,15614,15617,15620,15623,15663,15666,15669],{"className":15539},[399],[63,15541],{"className":15542},[399],[63,15544],{"className":15545,"style":777},[776],[63,15547,204],{"className":15548},[781],[63,15550],{"className":15551,"style":777},[776],[63,15553,14525],{"className":15554},[399],[63,15556],{"className":15557,"style":1177},[776],[63,15559,15561],{"className":15560},[1212,14383],[63,15562,15564,15599],{"className":15563},[411,412],[63,15565,15567,15596],{"className":15566},[416],[63,15568,15571,15584],{"className":15569,"style":15570},[420],"height:1.05em;",[63,15572,15574,15578],{"style":15573},"top:-1.9em;margin-left:0em;",[63,15575],{"className":15576,"style":15577},[428],"height:3.05em;",[63,15579,15581],{"className":15580},[433,434,435,436],[63,15582,13274],{"className":15583,"style":13290},[399,472,436],[63,15585,15587,15590],{"style":15586},"top:-3.05em;",[63,15588],{"className":15589,"style":15577},[428],[63,15591,15592],{},[63,15593,1874],{"className":15594},[1212,1958,15595],"large-op",[63,15597,507],{"className":15598},[506],[63,15600,15602],{"className":15601},[416],[63,15603,15606],{"className":15604,"style":15605},[420],"height:1.25em;",[63,15607],{},[63,15609],{"className":15610,"style":1177},[776],[63,15612,747],{"className":15613,"style":14254},[399,472],[63,15615,1078],{"className":15616},[403],[63,15618,13274],{"className":15619,"style":13290},[399,472],[63,15621,1092],{"className":15622},[593],[63,15624,15626,15629],{"className":15625},[399],[63,15627,5685],{"className":15628,"style":3251},[399,472],[63,15630,15632],{"className":15631},[476],[63,15633,15635,15655],{"className":15634},[411,412],[63,15636,15638,15652],{"className":15637},[416],[63,15639,15641],{"className":15640,"style":1532},[420],[63,15642,15643,15646],{"style":7483},[63,15644],{"className":15645,"style":817},[428],[63,15647,15649],{"className":15648},[433,434,435,436],[63,15650,14101],{"className":15651,"style":3276},[399,472,436],[63,15653,507],{"className":15654},[506],[63,15656,15658],{"className":15657},[416],[63,15659,15661],{"className":15660,"style":1554},[420],[63,15662],{},[63,15664,1078],{"className":15665},[403],[63,15667,13274],{"className":15668,"style":13290},[399,472],[63,15670,1092],{"className":15671},[593],[63,15673,15674,15677],{"style":15437},[63,15675],{"className":15676,"style":15315},[428],[63,15678,15680,15683,15686,15689,15692,15736,15739,15742,15745,15748,15751,15754,15794,15797,15800],{"className":15679},[399],[63,15681],{"className":15682},[399],[63,15684],{"className":15685,"style":777},[776],[63,15687,204],{"className":15688},[781],[63,15690],{"className":15691,"style":777},[776],[63,15693,15695],{"className":15694},[1212,14383],[63,15696,15698,15728],{"className":15697},[411,412],[63,15699,15701,15725],{"className":15700},[416],[63,15702,15704,15715],{"className":15703,"style":15570},[420],[63,15705,15706,15709],{"style":15573},[63,15707],{"className":15708,"style":15577},[428],[63,15710,15712],{"className":15711},[433,434,435,436],[63,15713,13274],{"className":15714,"style":13290},[399,472,436],[63,15716,15717,15720],{"style":15586},[63,15718],{"className":15719,"style":15577},[428],[63,15721,15722],{},[63,15723,1874],{"className":15724},[1212,1958,15595],[63,15726,507],{"className":15727},[506],[63,15729,15731],{"className":15730},[416],[63,15732,15734],{"className":15733,"style":15605},[420],[63,15735],{},[63,15737],{"className":15738,"style":1177},[776],[63,15740,747],{"className":15741,"style":14254},[399,472],[63,15743,1078],{"className":15744},[403],[63,15746,13274],{"className":15747,"style":13290},[399,472],[63,15749,1092],{"className":15750},[593],[63,15752,14525],{"className":15753},[399],[63,15755,15757,15760],{"className":15756},[399],[63,15758,5685],{"className":15759,"style":3251},[399,472],[63,15761,15763],{"className":15762},[476],[63,15764,15766,15786],{"className":15765},[411,412],[63,15767,15769,15783],{"className":15768},[416],[63,15770,15772],{"className":15771,"style":1532},[420],[63,15773,15774,15777],{"style":7483},[63,15775],{"className":15776,"style":817},[428],[63,15778,15780],{"className":15779},[433,434,435,436],[63,15781,14101],{"className":15782,"style":3276},[399,472,436],[63,15784,507],{"className":15785},[506],[63,15787,15789],{"className":15788},[416],[63,15790,15792],{"className":15791,"style":1554},[420],[63,15793],{},[63,15795,1078],{"className":15796},[403],[63,15798,13274],{"className":15799,"style":13290},[399,472],[63,15801,1092],{"className":15802},[593],[63,15804,15805,15808],{"style":15446},[63,15806],{"className":15807,"style":15315},[428],[63,15809,15811,15814,15817,15820,15823,15867,15870,15873,15876,15879,15882,15885,15925,15928,15931,15934,15937,15940,15943],{"className":15810},[399],[63,15812],{"className":15813},[399],[63,15815],{"className":15816,"style":777},[776],[63,15818,204],{"className":15819},[781],[63,15821],{"className":15822,"style":777},[776],[63,15824,15826],{"className":15825},[1212,14383],[63,15827,15829,15859],{"className":15828},[411,412],[63,15830,15832,15856],{"className":15831},[416],[63,15833,15835,15846],{"className":15834,"style":15570},[420],[63,15836,15837,15840],{"style":15573},[63,15838],{"className":15839,"style":15577},[428],[63,15841,15843],{"className":15842},[433,434,435,436],[63,15844,13274],{"className":15845,"style":13290},[399,472,436],[63,15847,15848,15851],{"style":15586},[63,15849],{"className":15850,"style":15577},[428],[63,15852,15853],{},[63,15854,1874],{"className":15855},[1212,1958,15595],[63,15857,507],{"className":15858},[506],[63,15860,15862],{"className":15861},[416],[63,15863,15865],{"className":15864,"style":15605},[420],[63,15866],{},[63,15868],{"className":15869,"style":1177},[776],[63,15871,747],{"className":15872,"style":14254},[399,472],[63,15874,1078],{"className":15875},[403],[63,15877,13274],{"className":15878,"style":13290},[399,472],[63,15880,1092],{"className":15881},[593],[63,15883,14525],{"className":15884},[399],[63,15886,15888,15891],{"className":15887},[399],[63,15889,5685],{"className":15890,"style":3251},[399,472],[63,15892,15894],{"className":15893},[476],[63,15895,15897,15917],{"className":15896},[411,412],[63,15898,15900,15914],{"className":15899},[416],[63,15901,15903],{"className":15902,"style":1532},[420],[63,15904,15905,15908],{"style":7483},[63,15906],{"className":15907,"style":817},[428],[63,15909,15911],{"className":15910},[433,434,435,436],[63,15912,14101],{"className":15913,"style":3276},[399,472,436],[63,15915,507],{"className":15916},[506],[63,15918,15920],{"className":15919},[416],[63,15921,15923],{"className":15922,"style":1554},[420],[63,15924],{},[63,15926,1078],{"className":15927},[403],[63,15929,13274],{"className":15930,"style":13290},[399,472],[63,15932,1092],{"className":15933},[593],[63,15935],{"className":15936,"style":1920},[776],[63,15938,1864],{"className":15939},[830],[63,15941],{"className":15942,"style":1920},[776],[63,15944,15946,15949,16098],{"className":15945},[399],[63,15947],{"className":15948},[403,404],[63,15950,15952],{"className":15951},[358],[63,15953,15955,16089],{"className":15954},[411,412],[63,15956,15958,16086],{"className":15957},[416],[63,15959,15962,16020,16028],{"className":15960,"style":15961},[420],"height:1.427em;",[63,15963,15965,15968],{"style":15964},"top:-2.314em;",[63,15966],{"className":15967,"style":429},[428],[63,15969,15971,16011,16014,16017],{"className":15970},[399],[63,15972,15974,15977],{"className":15973},[399],[63,15975,5685],{"className":15976,"style":3251},[399,472],[63,15978,15980],{"className":15979},[476],[63,15981,15983,16003],{"className":15982},[411,412],[63,15984,15986,16000],{"className":15985},[416],[63,15987,15989],{"className":15988,"style":1532},[420],[63,15990,15991,15994],{"style":7483},[63,15992],{"className":15993,"style":817},[428],[63,15995,15997],{"className":15996},[433,434,435,436],[63,15998,14101],{"className":15999,"style":3276},[399,472,436],[63,16001,507],{"className":16002},[506],[63,16004,16006],{"className":16005},[416],[63,16007,16009],{"className":16008,"style":1554},[420],[63,16010],{},[63,16012,1078],{"className":16013},[403],[63,16015,13274],{"className":16016,"style":13290},[399,472],[63,16018,1092],{"className":16019},[593],[63,16021,16022,16025],{"style":554},[63,16023],{"className":16024,"style":429},[428],[63,16026],{"className":16027,"style":562},[561],[63,16029,16031,16034],{"style":16030},"top:-3.677em;",[63,16032],{"className":16033,"style":429},[428],[63,16035,16037,16077,16080,16083],{"className":16036},[399],[63,16038,16040,16043],{"className":16039},[399],[63,16041,5685],{"className":16042,"style":3251},[399,472],[63,16044,16046],{"className":16045},[476],[63,16047,16049,16069],{"className":16048},[411,412],[63,16050,16052,16066],{"className":16051},[416],[63,16053,16055],{"className":16054,"style":1532},[420],[63,16056,16057,16060],{"style":7483},[63,16058],{"className":16059,"style":817},[428],[63,16061,16063],{"className":16062},[433,434,435,436],[63,16064,14101],{"className":16065,"style":3276},[399,472,436],[63,16067,507],{"className":16068},[506],[63,16070,16072],{"className":16071},[416],[63,16073,16075],{"className":16074,"style":1554},[420],[63,16076],{},[63,16078,1078],{"className":16079},[403],[63,16081,13274],{"className":16082,"style":13290},[399,472],[63,16084,1092],{"className":16085},[593],[63,16087,507],{"className":16088},[506],[63,16090,16092],{"className":16091},[416],[63,16093,16096],{"className":16094,"style":16095},[420],"height:0.936em;",[63,16097],{},[63,16099],{"className":16100},[593,404],[63,16102,16103,16106],{"style":15455},[63,16104],{"className":16105,"style":15315},[428],[63,16107,16109,16112,16115,16118,16121,16165,16168,16208,16211,16214,16217,16220,16223,16226,16229],{"className":16108},[399],[63,16110],{"className":16111},[399],[63,16113],{"className":16114,"style":777},[776],[63,16116,204],{"className":16117},[781],[63,16119],{"className":16120,"style":777},[776],[63,16122,16124],{"className":16123},[1212,14383],[63,16125,16127,16157],{"className":16126},[411,412],[63,16128,16130,16154],{"className":16129},[416],[63,16131,16133,16144],{"className":16132,"style":15570},[420],[63,16134,16135,16138],{"style":15573},[63,16136],{"className":16137,"style":15577},[428],[63,16139,16141],{"className":16140},[433,434,435,436],[63,16142,13274],{"className":16143,"style":13290},[399,472,436],[63,16145,16146,16149],{"style":15586},[63,16147],{"className":16148,"style":15577},[428],[63,16150,16151],{},[63,16152,1874],{"className":16153},[1212,1958,15595],[63,16155,507],{"className":16156},[506],[63,16158,16160],{"className":16159},[416],[63,16161,16163],{"className":16162,"style":15605},[420],[63,16164],{},[63,16166],{"className":16167,"style":1177},[776],[63,16169,16171,16174],{"className":16170},[399],[63,16172,5685],{"className":16173,"style":3251},[399,472],[63,16175,16177],{"className":16176},[476],[63,16178,16180,16200],{"className":16179},[411,412],[63,16181,16183,16197],{"className":16182},[416],[63,16184,16186],{"className":16185,"style":1532},[420],[63,16187,16188,16191],{"style":7483},[63,16189],{"className":16190,"style":817},[428],[63,16192,16194],{"className":16193},[433,434,435,436],[63,16195,14101],{"className":16196,"style":3276},[399,472,436],[63,16198,507],{"className":16199},[506],[63,16201,16203],{"className":16202},[416],[63,16204,16206],{"className":16205,"style":1554},[420],[63,16207],{},[63,16209,1078],{"className":16210},[403],[63,16212,13274],{"className":16213,"style":13290},[399,472],[63,16215,1092],{"className":16216},[593],[63,16218,747],{"className":16219,"style":14254},[399,472],[63,16221,1078],{"className":16222},[403],[63,16224,13274],{"className":16225,"style":13290},[399,472],[63,16227,1092],{"className":16228},[593],[63,16230,16232,16235,16383],{"className":16231},[399],[63,16233],{"className":16234},[403,404],[63,16236,16238],{"className":16237},[358],[63,16239,16241,16375],{"className":16240},[411,412],[63,16242,16244,16372],{"className":16243},[416],[63,16245,16247,16304,16312],{"className":16246,"style":15961},[420],[63,16248,16249,16252],{"style":15964},[63,16250],{"className":16251,"style":429},[428],[63,16253,16255,16295,16298,16301],{"className":16254},[399],[63,16256,16258,16261],{"className":16257},[399],[63,16259,5685],{"className":16260,"style":3251},[399,472],[63,16262,16264],{"className":16263},[476],[63,16265,16267,16287],{"className":16266},[411,412],[63,16268,16270,16284],{"className":16269},[416],[63,16271,16273],{"className":16272,"style":1532},[420],[63,16274,16275,16278],{"style":7483},[63,16276],{"className":16277,"style":817},[428],[63,16279,16281],{"className":16280},[433,434,435,436],[63,16282,14101],{"className":16283,"style":3276},[399,472,436],[63,16285,507],{"className":16286},[506],[63,16288,16290],{"className":16289},[416],[63,16291,16293],{"className":16292,"style":1554},[420],[63,16294],{},[63,16296,1078],{"className":16297},[403],[63,16299,13274],{"className":16300,"style":13290},[399,472],[63,16302,1092],{"className":16303},[593],[63,16305,16306,16309],{"style":554},[63,16307],{"className":16308,"style":429},[428],[63,16310],{"className":16311,"style":562},[561],[63,16313,16314,16317],{"style":16030},[63,16315],{"className":16316,"style":429},[428],[63,16318,16320,16323,16363,16366,16369],{"className":16319},[399],[63,16321,14525],{"className":16322},[399],[63,16324,16326,16329],{"className":16325},[399],[63,16327,5685],{"className":16328,"style":3251},[399,472],[63,16330,16332],{"className":16331},[476],[63,16333,16335,16355],{"className":16334},[411,412],[63,16336,16338,16352],{"className":16337},[416],[63,16339,16341],{"className":16340,"style":1532},[420],[63,16342,16343,16346],{"style":7483},[63,16344],{"className":16345,"style":817},[428],[63,16347,16349],{"className":16348},[433,434,435,436],[63,16350,14101],{"className":16351,"style":3276},[399,472,436],[63,16353,507],{"className":16354},[506],[63,16356,16358],{"className":16357},[416],[63,16359,16361],{"className":16360,"style":1554},[420],[63,16362],{},[63,16364,1078],{"className":16365},[403],[63,16367,13274],{"className":16368,"style":13290},[399,472],[63,16370,1092],{"className":16371},[593],[63,16373,507],{"className":16374},[506],[63,16376,16378],{"className":16377},[416],[63,16379,16381],{"className":16380,"style":16095},[420],[63,16382],{},[63,16384],{"className":16385},[593,404],[63,16387,16388,16391],{"style":15464},[63,16389],{"className":16390,"style":15315},[428],[63,16392,16394,16397,16400,16403,16406,16471,16474,16542,16545,16548,16551,16582,16585],{"className":16393},[399],[63,16395],{"className":16396},[399],[63,16398],{"className":16399,"style":777},[776],[63,16401,14820],{"className":16402},[781],[63,16404],{"className":16405,"style":777},[776],[63,16407,16409,16412,16468],{"className":16408},[399],[63,16410],{"className":16411},[403,404],[63,16413,16415],{"className":16414},[358],[63,16416,16418,16459],{"className":16417},[411,412],[63,16419,16421,16456],{"className":16420},[416],[63,16422,16425,16437,16445],{"className":16423,"style":16424},[420],"height:1.3214em;",[63,16426,16427,16430],{"style":15964},[63,16428],{"className":16429,"style":429},[428],[63,16431,16433],{"className":16432},[399],[63,16434,7342],{"className":16435,"style":16436},[399,472],"margin-right:0.10903em;",[63,16438,16439,16442],{"style":554},[63,16440],{"className":16441,"style":429},[428],[63,16443],{"className":16444,"style":562},[561],[63,16446,16447,16450],{"style":16030},[63,16448],{"className":16449,"style":429},[428],[63,16451,16453],{"className":16452},[399],[63,16454,363],{"className":16455},[399],[63,16457,507],{"className":16458},[506],[63,16460,16462],{"className":16461},[416],[63,16463,16466],{"className":16464,"style":16465},[420],"height:0.686em;",[63,16467],{},[63,16469],{"className":16470},[593,404],[63,16472],{"className":16473,"style":1177},[776],[63,16475,16477],{"className":16476},[1212,14383],[63,16478,16480,16533],{"className":16479},[411,412],[63,16481,16483,16530],{"className":16482},[416],[63,16484,16487,16508,16518],{"className":16485,"style":16486},[420],"height:1.8283em;",[63,16488,16490,16493],{"style":16489},"top:-1.8829em;margin-left:0em;",[63,16491],{"className":16492,"style":15577},[428],[63,16494,16496],{"className":16495},[433,434,435,436],[63,16497,16499,16502,16505],{"className":16498},[399,436],[63,16500,612],{"className":16501},[399,472,436],[63,16503,204],{"className":16504},[781,436],[63,16506,363],{"className":16507},[399,436],[63,16509,16510,16513],{"style":15586},[63,16511],{"className":16512,"style":15577},[428],[63,16514,16515],{},[63,16516,1874],{"className":16517},[1212,1958,15595],[63,16519,16521,16524],{"style":16520},"top:-4.3em;margin-left:0em;",[63,16522],{"className":16523,"style":15577},[428],[63,16525,16527],{"className":16526},[433,434,435,436],[63,16528,7342],{"className":16529,"style":16436},[399,472,436],[63,16531,507],{"className":16532},[506],[63,16534,16536],{"className":16535},[416],[63,16537,16540],{"className":16538,"style":16539},[420],"height:1.2671em;",[63,16541],{},[63,16543],{"className":16544,"style":1177},[776],[63,16546,747],{"className":16547,"style":14254},[399,472],[63,16549,1078],{"className":16550},[403],[63,16552,16554,16557],{"className":16553},[399],[63,16555,13274],{"className":16556,"style":13290},[399,472],[63,16558,16560],{"className":16559},[476],[63,16561,16563],{"className":16562},[411],[63,16564,16566],{"className":16565},[416],[63,16567,16570],{"className":16568,"style":16569},[420],"height:0.7144em;",[63,16571,16573,16576],{"style":16572},"top:-3.113em;margin-right:0.05em;",[63,16574],{"className":16575,"style":817},[428],[63,16577,16579],{"className":16578},[433,434,435,436],[63,16580,612],{"className":16581},[399,472,436],[63,16583,1092],{"className":16584},[593],[63,16586,16588,16591,16794],{"className":16587},[399],[63,16589],{"className":16590},[403,404],[63,16592,16594],{"className":16593},[358],[63,16595,16597,16786],{"className":16596},[411,412],[63,16598,16600,16783],{"className":16599},[416],[63,16601,16603,16688,16696],{"className":16602,"style":15961},[420],[63,16604,16605,16608],{"style":15964},[63,16606],{"className":16607,"style":429},[428],[63,16609,16611,16651,16654,16685],{"className":16610},[399],[63,16612,16614,16617],{"className":16613},[399],[63,16615,5685],{"className":16616,"style":3251},[399,472],[63,16618,16620],{"className":16619},[476],[63,16621,16623,16643],{"className":16622},[411,412],[63,16624,16626,16640],{"className":16625},[416],[63,16627,16629],{"className":16628,"style":1532},[420],[63,16630,16631,16634],{"style":7483},[63,16632],{"className":16633,"style":817},[428],[63,16635,16637],{"className":16636},[433,434,435,436],[63,16638,14101],{"className":16639,"style":3276},[399,472,436],[63,16641,507],{"className":16642},[506],[63,16644,16646],{"className":16645},[416],[63,16647,16649],{"className":16648,"style":1554},[420],[63,16650],{},[63,16652,1078],{"className":16653},[403],[63,16655,16657,16660],{"className":16656},[399],[63,16658,13274],{"className":16659,"style":13290},[399,472],[63,16661,16663],{"className":16662},[476],[63,16664,16666],{"className":16665},[411],[63,16667,16669],{"className":16668},[416],[63,16670,16673],{"className":16671,"style":16672},[420],"height:0.5904em;",[63,16674,16676,16679],{"style":16675},"top:-2.989em;margin-right:0.05em;",[63,16677],{"className":16678,"style":817},[428],[63,16680,16682],{"className":16681},[433,434,435,436],[63,16683,612],{"className":16684},[399,472,436],[63,16686,1092],{"className":16687},[593],[63,16689,16690,16693],{"style":554},[63,16691],{"className":16692,"style":429},[428],[63,16694],{"className":16695,"style":562},[561],[63,16697,16698,16701],{"style":16030},[63,16699],{"className":16700,"style":429},[428],[63,16702,16704,16707,16747,16750,16780],{"className":16703},[399],[63,16705,14525],{"className":16706},[399],[63,16708,16710,16713],{"className":16709},[399],[63,16711,14],{"className":16712},[399,472],[63,16714,16716],{"className":16715},[476],[63,16717,16719,16739],{"className":16718},[411,412],[63,16720,16722,16736],{"className":16721},[416],[63,16723,16725],{"className":16724,"style":1532},[420],[63,16726,16727,16730],{"style":1535},[63,16728],{"className":16729,"style":817},[428],[63,16731,16733],{"className":16732},[433,434,435,436],[63,16734,14101],{"className":16735,"style":3276},[399,472,436],[63,16737,507],{"className":16738},[506],[63,16740,16742],{"className":16741},[416],[63,16743,16745],{"className":16744,"style":1554},[420],[63,16746],{},[63,16748,1078],{"className":16749},[403],[63,16751,16753,16756],{"className":16752},[399],[63,16754,13274],{"className":16755,"style":13290},[399,472],[63,16757,16759],{"className":16758},[476],[63,16760,16762],{"className":16761},[411],[63,16763,16765],{"className":16764},[416],[63,16766,16769],{"className":16767,"style":16768},[420],"height:0.6644em;",[63,16770,16771,16774],{"style":813},[63,16772],{"className":16773,"style":817},[428],[63,16775,16777],{"className":16776},[433,434,435,436],[63,16778,612],{"className":16779},[399,472,436],[63,16781,1092],{"className":16782},[593],[63,16784,507],{"className":16785},[506],[63,16787,16789],{"className":16788},[416],[63,16790,16792],{"className":16791,"style":16095},[420],[63,16793],{},[63,16795],{"className":16796},[593,404],[63,16798,16799,16802],{"style":15473},[63,16800],{"className":16801,"style":15315},[428],[63,16803,16805,16808,16811,16814,16817,16879,16882,16946,16949,16952,16955,16984,16987,16990,16993,17001,17004,17044,17047,17076],{"className":16804},[399],[63,16806],{"className":16807},[399],[63,16809],{"className":16810,"style":777},[776],[63,16812,204],{"className":16813},[781],[63,16815],{"className":16816,"style":777},[776],[63,16818,16820,16823,16876],{"className":16819},[399],[63,16821],{"className":16822},[403,404],[63,16824,16826],{"className":16825},[358],[63,16827,16829,16868],{"className":16828},[411,412],[63,16830,16832,16865],{"className":16831},[416],[63,16833,16835,16846,16854],{"className":16834,"style":16424},[420],[63,16836,16837,16840],{"style":15964},[63,16838],{"className":16839,"style":429},[428],[63,16841,16843],{"className":16842},[399],[63,16844,7342],{"className":16845,"style":16436},[399,472],[63,16847,16848,16851],{"style":554},[63,16849],{"className":16850,"style":429},[428],[63,16852],{"className":16853,"style":562},[561],[63,16855,16856,16859],{"style":16030},[63,16857],{"className":16858,"style":429},[428],[63,16860,16862],{"className":16861},[399],[63,16863,363],{"className":16864},[399],[63,16866,507],{"className":16867},[506],[63,16869,16871],{"className":16870},[416],[63,16872,16874],{"className":16873,"style":16465},[420],[63,16875],{},[63,16877],{"className":16878},[593,404],[63,16880],{"className":16881,"style":1177},[776],[63,16883,16885],{"className":16884},[1212,14383],[63,16886,16888,16938],{"className":16887},[411,412],[63,16889,16891,16935],{"className":16890},[416],[63,16892,16894,16914,16924],{"className":16893,"style":16486},[420],[63,16895,16896,16899],{"style":16489},[63,16897],{"className":16898,"style":15577},[428],[63,16900,16902],{"className":16901},[433,434,435,436],[63,16903,16905,16908,16911],{"className":16904},[399,436],[63,16906,612],{"className":16907},[399,472,436],[63,16909,204],{"className":16910},[781,436],[63,16912,363],{"className":16913},[399,436],[63,16915,16916,16919],{"style":15586},[63,16917],{"className":16918,"style":15577},[428],[63,16920,16921],{},[63,16922,1874],{"className":16923},[1212,1958,15595],[63,16925,16926,16929],{"style":16520},[63,16927],{"className":16928,"style":15577},[428],[63,16930,16932],{"className":16931},[433,434,435,436],[63,16933,7342],{"className":16934,"style":16436},[399,472,436],[63,16936,507],{"className":16937},[506],[63,16939,16941],{"className":16940},[416],[63,16942,16944],{"className":16943,"style":16539},[420],[63,16945],{},[63,16947],{"className":16948,"style":1177},[776],[63,16950,747],{"className":16951,"style":14254},[399,472],[63,16953,1078],{"className":16954},[403],[63,16956,16958,16961],{"className":16957},[399],[63,16959,13274],{"className":16960,"style":13290},[399,472],[63,16962,16964],{"className":16963},[476],[63,16965,16967],{"className":16966},[411],[63,16968,16970],{"className":16969},[416],[63,16971,16973],{"className":16972,"style":16569},[420],[63,16974,16975,16978],{"style":16572},[63,16976],{"className":16977,"style":817},[428],[63,16979,16981],{"className":16980},[433,434,435,436],[63,16982,612],{"className":16983},[399,472,436],[63,16985,1092],{"className":16986},[593],[63,16988,14525],{"className":16989},[399],[63,16991],{"className":16992,"style":1177},[776],[63,16994,16996,16997],{"className":16995},[1212],"lo",[63,16998,17000],{"style":16999},"margin-right:0.01389em;","g",[63,17002],{"className":17003,"style":1177},[776],[63,17005,17007,17010],{"className":17006},[399],[63,17008,5685],{"className":17009,"style":3251},[399,472],[63,17011,17013],{"className":17012},[476],[63,17014,17016,17036],{"className":17015},[411,412],[63,17017,17019,17033],{"className":17018},[416],[63,17020,17022],{"className":17021,"style":1532},[420],[63,17023,17024,17027],{"style":7483},[63,17025],{"className":17026,"style":817},[428],[63,17028,17030],{"className":17029},[433,434,435,436],[63,17031,14101],{"className":17032,"style":3276},[399,472,436],[63,17034,507],{"className":17035},[506],[63,17037,17039],{"className":17038},[416],[63,17040,17042],{"className":17041,"style":1554},[420],[63,17043],{},[63,17045,1078],{"className":17046},[403],[63,17048,17050,17053],{"className":17049},[399],[63,17051,13274],{"className":17052,"style":13290},[399,472],[63,17054,17056],{"className":17055},[476],[63,17057,17059],{"className":17058},[411],[63,17060,17062],{"className":17061},[416],[63,17063,17065],{"className":17064,"style":16569},[420],[63,17066,17067,17070],{"style":16572},[63,17068],{"className":17069,"style":817},[428],[63,17071,17073],{"className":17072},[433,434,435,436],[63,17074,612],{"className":17075},[399,472,436],[63,17077,1092],{"className":17078},[593],[63,17080,17081,17084],{"style":15482},[63,17082],{"className":17083,"style":15315},[428],[63,17085,17087,17090,17093,17096,17099,17161,17164,17228,17231,17234,17237,17266,17269,17272,17275,17280,17283,17390,17393,17433,17436,17490,17493,17544],{"className":17086},[399],[63,17088],{"className":17089},[399],[63,17091],{"className":17092,"style":777},[776],[63,17094,204],{"className":17095},[781],[63,17097],{"className":17098,"style":777},[776],[63,17100,17102,17105,17158],{"className":17101},[399],[63,17103],{"className":17104},[403,404],[63,17106,17108],{"className":17107},[358],[63,17109,17111,17150],{"className":17110},[411,412],[63,17112,17114,17147],{"className":17113},[416],[63,17115,17117,17128,17136],{"className":17116,"style":16424},[420],[63,17118,17119,17122],{"style":15964},[63,17120],{"className":17121,"style":429},[428],[63,17123,17125],{"className":17124},[399],[63,17126,7342],{"className":17127,"style":16436},[399,472],[63,17129,17130,17133],{"style":554},[63,17131],{"className":17132,"style":429},[428],[63,17134],{"className":17135,"style":562},[561],[63,17137,17138,17141],{"style":16030},[63,17139],{"className":17140,"style":429},[428],[63,17142,17144],{"className":17143},[399],[63,17145,363],{"className":17146},[399],[63,17148,507],{"className":17149},[506],[63,17151,17153],{"className":17152},[416],[63,17154,17156],{"className":17155,"style":16465},[420],[63,17157],{},[63,17159],{"className":17160},[593,404],[63,17162],{"className":17163,"style":1177},[776],[63,17165,17167],{"className":17166},[1212,14383],[63,17168,17170,17220],{"className":17169},[411,412],[63,17171,17173,17217],{"className":17172},[416],[63,17174,17176,17196,17206],{"className":17175,"style":16486},[420],[63,17177,17178,17181],{"style":16489},[63,17179],{"className":17180,"style":15577},[428],[63,17182,17184],{"className":17183},[433,434,435,436],[63,17185,17187,17190,17193],{"className":17186},[399,436],[63,17188,612],{"className":17189},[399,472,436],[63,17191,204],{"className":17192},[781,436],[63,17194,363],{"className":17195},[399,436],[63,17197,17198,17201],{"style":15586},[63,17199],{"className":17200,"style":15577},[428],[63,17202,17203],{},[63,17204,1874],{"className":17205},[1212,1958,15595],[63,17207,17208,17211],{"style":16520},[63,17209],{"className":17210,"style":15577},[428],[63,17212,17214],{"className":17213},[433,434,435,436],[63,17215,7342],{"className":17216,"style":16436},[399,472,436],[63,17218,507],{"className":17219},[506],[63,17221,17223],{"className":17222},[416],[63,17224,17226],{"className":17225,"style":16539},[420],[63,17227],{},[63,17229],{"className":17230,"style":1177},[776],[63,17232,747],{"className":17233,"style":14254},[399,472],[63,17235,1078],{"className":17236},[403],[63,17238,17240,17243],{"className":17239},[399],[63,17241,13274],{"className":17242,"style":13290},[399,472],[63,17244,17246],{"className":17245},[476],[63,17247,17249],{"className":17248},[411],[63,17250,17252],{"className":17251},[416],[63,17253,17255],{"className":17254,"style":16569},[420],[63,17256,17257,17260],{"style":16572},[63,17258],{"className":17259,"style":817},[428],[63,17261,17263],{"className":17262},[433,434,435,436],[63,17264,612],{"className":17265},[399,472,436],[63,17267,1092],{"className":17268},[593],[63,17270,14525],{"className":17271},[399],[63,17273],{"className":17274,"style":1177},[776],[63,17276,16996,17278],{"className":17277},[1212],[63,17279,17000],{"style":16999},[63,17281],{"className":17282,"style":1177},[776],[63,17284,17286],{"className":17285},[1212,14383],[63,17287,17289,17382],{"className":17288},[411,412],[63,17290,17292,17379],{"className":17291},[416],[63,17293,17296,17316,17326],{"className":17294,"style":17295},[420],"height:1.8394em;",[63,17297,17298,17301],{"style":16489},[63,17299],{"className":17300,"style":15577},[428],[63,17302,17304],{"className":17303},[433,434,435,436],[63,17305,17307,17310,17313],{"className":17306},[399,436],[63,17308,1057],{"className":17309},[399,472,436],[63,17311,204],{"className":17312},[781,436],[63,17314,363],{"className":17315},[399,436],[63,17317,17318,17321],{"style":15586},[63,17319],{"className":17320,"style":15577},[428],[63,17322,17323],{},[63,17324,15029],{"className":17325},[1212,1958,15595],[63,17327,17329,17332],{"style":17328},"top:-4.3111em;margin-left:0em;",[63,17330],{"className":17331,"style":15577},[428],[63,17333,17335],{"className":17334},[433,434,435,436],[63,17336,17338],{"className":17337},[399,436],[63,17339,17341,17344],{"className":17340},[399,436],[63,17342,13358],{"className":17343,"style":3251},[399,472,436],[63,17345,17347],{"className":17346},[476],[63,17348,17350,17371],{"className":17349},[411,412],[63,17351,17353,17368],{"className":17352},[416],[63,17354,17356],{"className":17355,"style":4367},[420],[63,17357,17359,17362],{"style":17358},"top:-2.357em;margin-left:-0.1389em;margin-right:0.0714em;",[63,17360],{"className":17361,"style":493},[428],[63,17363,17365],{"className":17364},[433,497,498,436],[63,17366,612],{"className":17367},[399,472,436],[63,17369,507],{"className":17370},[506],[63,17372,17374],{"className":17373},[416],[63,17375,17377],{"className":17376,"style":4389},[420],[63,17378],{},[63,17380,507],{"className":17381},[506],[63,17383,17385],{"className":17384},[416],[63,17386,17388],{"className":17387,"style":16539},[420],[63,17389],{},[63,17391],{"className":17392,"style":1177},[776],[63,17394,17396,17399],{"className":17395},[399],[63,17397,5685],{"className":17398,"style":3251},[399,472],[63,17400,17402],{"className":17401},[476],[63,17403,17405,17425],{"className":17404},[411,412],[63,17406,17408,17422],{"className":17407},[416],[63,17409,17411],{"className":17410,"style":1532},[420],[63,17412,17413,17416],{"style":7483},[63,17414],{"className":17415,"style":817},[428],[63,17417,17419],{"className":17418},[433,434,435,436],[63,17420,14101],{"className":17421,"style":3276},[399,472,436],[63,17423,507],{"className":17424},[506],[63,17426,17428],{"className":17427},[416],[63,17429,17431],{"className":17430,"style":1554},[420],[63,17432],{},[63,17434,1078],{"className":17435},[403],[63,17437,17439,17442],{"className":17438},[399],[63,17440,38],{"className":17441},[399,472],[63,17443,17445],{"className":17444},[476],[63,17446,17448,17481],{"className":17447},[411,412],[63,17449,17451,17478],{"className":17450},[416],[63,17452,17455,17467],{"className":17453,"style":17454},[420],"height:0.8436em;",[63,17456,17458,17461],{"style":17457},"top:-2.453em;margin-left:0em;margin-right:0.05em;",[63,17459],{"className":17460,"style":817},[428],[63,17462,17464],{"className":17463},[433,434,435,436],[63,17465,612],{"className":17466},[399,472,436],[63,17468,17469,17472],{"style":16572},[63,17470],{"className":17471,"style":817},[428],[63,17473,17475],{"className":17474},[433,434,435,436],[63,17476,1057],{"className":17477},[399,472,436],[63,17479,507],{"className":17480},[506],[63,17482,17484],{"className":17483},[416],[63,17485,17488],{"className":17486,"style":17487},[420],"height:0.247em;",[63,17489],{},[63,17491,13030],{"className":17492},[399],[63,17494,17496,17499],{"className":17495},[399],[63,17497,12890],{"className":17498},[399,472],[63,17500,17502],{"className":17501},[476],[63,17503,17505,17536],{"className":17504},[411,412],[63,17506,17508,17533],{"className":17507},[416],[63,17509,17511,17522],{"className":17510,"style":17454},[420],[63,17512,17513,17516],{"style":17457},[63,17514],{"className":17515,"style":817},[428],[63,17517,17519],{"className":17518},[433,434,435,436],[63,17520,612],{"className":17521},[399,472,436],[63,17523,17524,17527],{"style":16572},[63,17525],{"className":17526,"style":817},[428],[63,17528,17530],{"className":17529},[433,434,435,436],[63,17531,1057],{"className":17532},[399,472,436],[63,17534,507],{"className":17535},[506],[63,17537,17539],{"className":17538},[416],[63,17540,17542],{"className":17541,"style":17487},[420],[63,17543],{},[63,17545,1092],{"className":17546},[593],[63,17548,17549,17552],{"style":15491},[63,17550],{"className":17551,"style":15315},[428],[63,17553,17555,17558,17561,17564,17567,17629,17632,17696,17699,17702,17705,17734,17737,17740,17844,17847,17850,17853,17858,17861,17901,17904,17955,17958,18009],{"className":17554},[399],[63,17556],{"className":17557},[399],[63,17559],{"className":17560,"style":777},[776],[63,17562,204],{"className":17563},[781],[63,17565],{"className":17566,"style":777},[776],[63,17568,17570,17573,17626],{"className":17569},[399],[63,17571],{"className":17572},[403,404],[63,17574,17576],{"className":17575},[358],[63,17577,17579,17618],{"className":17578},[411,412],[63,17580,17582,17615],{"className":17581},[416],[63,17583,17585,17596,17604],{"className":17584,"style":16424},[420],[63,17586,17587,17590],{"style":15964},[63,17588],{"className":17589,"style":429},[428],[63,17591,17593],{"className":17592},[399],[63,17594,7342],{"className":17595,"style":16436},[399,472],[63,17597,17598,17601],{"style":554},[63,17599],{"className":17600,"style":429},[428],[63,17602],{"className":17603,"style":562},[561],[63,17605,17606,17609],{"style":16030},[63,17607],{"className":17608,"style":429},[428],[63,17610,17612],{"className":17611},[399],[63,17613,363],{"className":17614},[399],[63,17616,507],{"className":17617},[506],[63,17619,17621],{"className":17620},[416],[63,17622,17624],{"className":17623,"style":16465},[420],[63,17625],{},[63,17627],{"className":17628},[593,404],[63,17630],{"className":17631,"style":1177},[776],[63,17633,17635],{"className":17634},[1212,14383],[63,17636,17638,17688],{"className":17637},[411,412],[63,17639,17641,17685],{"className":17640},[416],[63,17642,17644,17664,17674],{"className":17643,"style":16486},[420],[63,17645,17646,17649],{"style":16489},[63,17647],{"className":17648,"style":15577},[428],[63,17650,17652],{"className":17651},[433,434,435,436],[63,17653,17655,17658,17661],{"className":17654},[399,436],[63,17656,612],{"className":17657},[399,472,436],[63,17659,204],{"className":17660},[781,436],[63,17662,363],{"className":17663},[399,436],[63,17665,17666,17669],{"style":15586},[63,17667],{"className":17668,"style":15577},[428],[63,17670,17671],{},[63,17672,1874],{"className":17673},[1212,1958,15595],[63,17675,17676,17679],{"style":16520},[63,17677],{"className":17678,"style":15577},[428],[63,17680,17682],{"className":17681},[433,434,435,436],[63,17683,7342],{"className":17684,"style":16436},[399,472,436],[63,17686,507],{"className":17687},[506],[63,17689,17691],{"className":17690},[416],[63,17692,17694],{"className":17693,"style":16539},[420],[63,17695],{},[63,17697],{"className":17698,"style":1177},[776],[63,17700,747],{"className":17701,"style":14254},[399,472],[63,17703,1078],{"className":17704},[403],[63,17706,17708,17711],{"className":17707},[399],[63,17709,13274],{"className":17710,"style":13290},[399,472],[63,17712,17714],{"className":17713},[476],[63,17715,17717],{"className":17716},[411],[63,17718,17720],{"className":17719},[416],[63,17721,17723],{"className":17722,"style":16569},[420],[63,17724,17725,17728],{"style":16572},[63,17726],{"className":17727,"style":817},[428],[63,17729,17731],{"className":17730},[433,434,435,436],[63,17732,612],{"className":17733},[399,472,436],[63,17735,1092],{"className":17736},[593],[63,17738],{"className":17739,"style":1177},[776],[63,17741,17743],{"className":17742},[1212,14383],[63,17744,17746,17836],{"className":17745},[411,412],[63,17747,17749,17833],{"className":17748},[416],[63,17750,17752,17772,17782],{"className":17751,"style":17295},[420],[63,17753,17754,17757],{"style":16489},[63,17755],{"className":17756,"style":15577},[428],[63,17758,17760],{"className":17759},[433,434,435,436],[63,17761,17763,17766,17769],{"className":17762},[399,436],[63,17764,1057],{"className":17765},[399,472,436],[63,17767,204],{"className":17768},[781,436],[63,17770,363],{"className":17771},[399,436],[63,17773,17774,17777],{"style":15586},[63,17775],{"className":17776,"style":15577},[428],[63,17778,17779],{},[63,17780,1874],{"className":17781},[1212,1958,15595],[63,17783,17784,17787],{"style":17328},[63,17785],{"className":17786,"style":15577},[428],[63,17788,17790],{"className":17789},[433,434,435,436],[63,17791,17793],{"className":17792},[399,436],[63,17794,17796,17799],{"className":17795},[399,436],[63,17797,13358],{"className":17798,"style":3251},[399,472,436],[63,17800,17802],{"className":17801},[476],[63,17803,17805,17825],{"className":17804},[411,412],[63,17806,17808,17822],{"className":17807},[416],[63,17809,17811],{"className":17810,"style":4367},[420],[63,17812,17813,17816],{"style":17358},[63,17814],{"className":17815,"style":493},[428],[63,17817,17819],{"className":17818},[433,497,498,436],[63,17820,612],{"className":17821},[399,472,436],[63,17823,507],{"className":17824},[506],[63,17826,17828],{"className":17827},[416],[63,17829,17831],{"className":17830,"style":4389},[420],[63,17832],{},[63,17834,507],{"className":17835},[506],[63,17837,17839],{"className":17838},[416],[63,17840,17842],{"className":17841,"style":16539},[420],[63,17843],{},[63,17845],{"className":17846,"style":1177},[776],[63,17848,14525],{"className":17849},[399],[63,17851],{"className":17852,"style":1177},[776],[63,17854,16996,17856],{"className":17855},[1212],[63,17857,17000],{"style":16999},[63,17859],{"className":17860,"style":1177},[776],[63,17862,17864,17867],{"className":17863},[399],[63,17865,5685],{"className":17866,"style":3251},[399,472],[63,17868,17870],{"className":17869},[476],[63,17871,17873,17893],{"className":17872},[411,412],[63,17874,17876,17890],{"className":17875},[416],[63,17877,17879],{"className":17878,"style":1532},[420],[63,17880,17881,17884],{"style":7483},[63,17882],{"className":17883,"style":817},[428],[63,17885,17887],{"className":17886},[433,434,435,436],[63,17888,14101],{"className":17889,"style":3276},[399,472,436],[63,17891,507],{"className":17892},[506],[63,17894,17896],{"className":17895},[416],[63,17897,17899],{"className":17898,"style":1554},[420],[63,17900],{},[63,17902,1078],{"className":17903},[403],[63,17905,17907,17910],{"className":17906},[399],[63,17908,38],{"className":17909},[399,472],[63,17911,17913],{"className":17912},[476],[63,17914,17916,17947],{"className":17915},[411,412],[63,17917,17919,17944],{"className":17918},[416],[63,17920,17922,17933],{"className":17921,"style":17454},[420],[63,17923,17924,17927],{"style":17457},[63,17925],{"className":17926,"style":817},[428],[63,17928,17930],{"className":17929},[433,434,435,436],[63,17931,612],{"className":17932},[399,472,436],[63,17934,17935,17938],{"style":16572},[63,17936],{"className":17937,"style":817},[428],[63,17939,17941],{"className":17940},[433,434,435,436],[63,17942,1057],{"className":17943},[399,472,436],[63,17945,507],{"className":17946},[506],[63,17948,17950],{"className":17949},[416],[63,17951,17953],{"className":17952,"style":17487},[420],[63,17954],{},[63,17956,13030],{"className":17957},[399],[63,17959,17961,17964],{"className":17960},[399],[63,17962,12890],{"className":17963},[399,472],[63,17965,17967],{"className":17966},[476],[63,17968,17970,18001],{"className":17969},[411,412],[63,17971,17973,17998],{"className":17972},[416],[63,17974,17976,17987],{"className":17975,"style":17454},[420],[63,17977,17978,17981],{"style":17457},[63,17979],{"className":17980,"style":817},[428],[63,17982,17984],{"className":17983},[433,434,435,436],[63,17985,612],{"className":17986},[399,472,436],[63,17988,17989,17992],{"style":16572},[63,17990],{"className":17991,"style":817},[428],[63,17993,17995],{"className":17994},[433,434,435,436],[63,17996,1057],{"className":17997},[399,472,436],[63,17999,507],{"className":18000},[506],[63,18002,18004],{"className":18003},[416],[63,18005,18007],{"className":18006,"style":17487},[420],[63,18008],{},[63,18010,1092],{"className":18011},[593],[63,18013,18014,18017],{"style":15500},[63,18015],{"className":18016,"style":15315},[428],[63,18018,18020,18023,18026,18029,18032,18094,18097,18161,18164,18268,18271,18274,18277,18306,18309,18312,18315,18320,18323,18363,18366,18417,18420,18471],{"className":18019},[399],[63,18021],{"className":18022},[399],[63,18024],{"className":18025,"style":777},[776],[63,18027,204],{"className":18028},[781],[63,18030],{"className":18031,"style":777},[776],[63,18033,18035,18038,18091],{"className":18034},[399],[63,18036],{"className":18037},[403,404],[63,18039,18041],{"className":18040},[358],[63,18042,18044,18083],{"className":18043},[411,412],[63,18045,18047,18080],{"className":18046},[416],[63,18048,18050,18061,18069],{"className":18049,"style":16424},[420],[63,18051,18052,18055],{"style":15964},[63,18053],{"className":18054,"style":429},[428],[63,18056,18058],{"className":18057},[399],[63,18059,7342],{"className":18060,"style":16436},[399,472],[63,18062,18063,18066],{"style":554},[63,18064],{"className":18065,"style":429},[428],[63,18067],{"className":18068,"style":562},[561],[63,18070,18071,18074],{"style":16030},[63,18072],{"className":18073,"style":429},[428],[63,18075,18077],{"className":18076},[399],[63,18078,363],{"className":18079},[399],[63,18081,507],{"className":18082},[506],[63,18084,18086],{"className":18085},[416],[63,18087,18089],{"className":18088,"style":16465},[420],[63,18090],{},[63,18092],{"className":18093},[593,404],[63,18095],{"className":18096,"style":1177},[776],[63,18098,18100],{"className":18099},[1212,14383],[63,18101,18103,18153],{"className":18102},[411,412],[63,18104,18106,18150],{"className":18105},[416],[63,18107,18109,18129,18139],{"className":18108,"style":16486},[420],[63,18110,18111,18114],{"style":16489},[63,18112],{"className":18113,"style":15577},[428],[63,18115,18117],{"className":18116},[433,434,435,436],[63,18118,18120,18123,18126],{"className":18119},[399,436],[63,18121,612],{"className":18122},[399,472,436],[63,18124,204],{"className":18125},[781,436],[63,18127,363],{"className":18128},[399,436],[63,18130,18131,18134],{"style":15586},[63,18132],{"className":18133,"style":15577},[428],[63,18135,18136],{},[63,18137,1874],{"className":18138},[1212,1958,15595],[63,18140,18141,18144],{"style":16520},[63,18142],{"className":18143,"style":15577},[428],[63,18145,18147],{"className":18146},[433,434,435,436],[63,18148,7342],{"className":18149,"style":16436},[399,472,436],[63,18151,507],{"className":18152},[506],[63,18154,18156],{"className":18155},[416],[63,18157,18159],{"className":18158,"style":16539},[420],[63,18160],{},[63,18162],{"className":18163,"style":1177},[776],[63,18165,18167],{"className":18166},[1212,14383],[63,18168,18170,18260],{"className":18169},[411,412],[63,18171,18173,18257],{"className":18172},[416],[63,18174,18176,18196,18206],{"className":18175,"style":17295},[420],[63,18177,18178,18181],{"style":16489},[63,18179],{"className":18180,"style":15577},[428],[63,18182,18184],{"className":18183},[433,434,435,436],[63,18185,18187,18190,18193],{"className":18186},[399,436],[63,18188,1057],{"className":18189},[399,472,436],[63,18191,204],{"className":18192},[781,436],[63,18194,363],{"className":18195},[399,436],[63,18197,18198,18201],{"style":15586},[63,18199],{"className":18200,"style":15577},[428],[63,18202,18203],{},[63,18204,1874],{"className":18205},[1212,1958,15595],[63,18207,18208,18211],{"style":17328},[63,18209],{"className":18210,"style":15577},[428],[63,18212,18214],{"className":18213},[433,434,435,436],[63,18215,18217],{"className":18216},[399,436],[63,18218,18220,18223],{"className":18219},[399,436],[63,18221,13358],{"className":18222,"style":3251},[399,472,436],[63,18224,18226],{"className":18225},[476],[63,18227,18229,18249],{"className":18228},[411,412],[63,18230,18232,18246],{"className":18231},[416],[63,18233,18235],{"className":18234,"style":4367},[420],[63,18236,18237,18240],{"style":17358},[63,18238],{"className":18239,"style":493},[428],[63,18241,18243],{"className":18242},[433,497,498,436],[63,18244,612],{"className":18245},[399,472,436],[63,18247,507],{"className":18248},[506],[63,18250,18252],{"className":18251},[416],[63,18253,18255],{"className":18254,"style":4389},[420],[63,18256],{},[63,18258,507],{"className":18259},[506],[63,18261,18263],{"className":18262},[416],[63,18264,18266],{"className":18265,"style":16539},[420],[63,18267],{},[63,18269],{"className":18270,"style":1177},[776],[63,18272,747],{"className":18273,"style":14254},[399,472],[63,18275,1078],{"className":18276},[403],[63,18278,18280,18283],{"className":18279},[399],[63,18281,13274],{"className":18282,"style":13290},[399,472],[63,18284,18286],{"className":18285},[476],[63,18287,18289],{"className":18288},[411],[63,18290,18292],{"className":18291},[416],[63,18293,18295],{"className":18294,"style":16569},[420],[63,18296,18297,18300],{"style":16572},[63,18298],{"className":18299,"style":817},[428],[63,18301,18303],{"className":18302},[433,434,435,436],[63,18304,612],{"className":18305},[399,472,436],[63,18307,1092],{"className":18308},[593],[63,18310,14525],{"className":18311},[399],[63,18313],{"className":18314,"style":1177},[776],[63,18316,16996,18318],{"className":18317},[1212],[63,18319,17000],{"style":16999},[63,18321],{"className":18322,"style":1177},[776],[63,18324,18326,18329],{"className":18325},[399],[63,18327,5685],{"className":18328,"style":3251},[399,472],[63,18330,18332],{"className":18331},[476],[63,18333,18335,18355],{"className":18334},[411,412],[63,18336,18338,18352],{"className":18337},[416],[63,18339,18341],{"className":18340,"style":1532},[420],[63,18342,18343,18346],{"style":7483},[63,18344],{"className":18345,"style":817},[428],[63,18347,18349],{"className":18348},[433,434,435,436],[63,18350,14101],{"className":18351,"style":3276},[399,472,436],[63,18353,507],{"className":18354},[506],[63,18356,18358],{"className":18357},[416],[63,18359,18361],{"className":18360,"style":1554},[420],[63,18362],{},[63,18364,1078],{"className":18365},[403],[63,18367,18369,18372],{"className":18368},[399],[63,18370,38],{"className":18371},[399,472],[63,18373,18375],{"className":18374},[476],[63,18376,18378,18409],{"className":18377},[411,412],[63,18379,18381,18406],{"className":18380},[416],[63,18382,18384,18395],{"className":18383,"style":17454},[420],[63,18385,18386,18389],{"style":17457},[63,18387],{"className":18388,"style":817},[428],[63,18390,18392],{"className":18391},[433,434,435,436],[63,18393,612],{"className":18394},[399,472,436],[63,18396,18397,18400],{"style":16572},[63,18398],{"className":18399,"style":817},[428],[63,18401,18403],{"className":18402},[433,434,435,436],[63,18404,1057],{"className":18405},[399,472,436],[63,18407,507],{"className":18408},[506],[63,18410,18412],{"className":18411},[416],[63,18413,18415],{"className":18414,"style":17487},[420],[63,18416],{},[63,18418,13030],{"className":18419},[399],[63,18421,18423,18426],{"className":18422},[399],[63,18424,12890],{"className":18425},[399,472],[63,18427,18429],{"className":18428},[476],[63,18430,18432,18463],{"className":18431},[411,412],[63,18433,18435,18460],{"className":18434},[416],[63,18436,18438,18449],{"className":18437,"style":17454},[420],[63,18439,18440,18443],{"style":17457},[63,18441],{"className":18442,"style":817},[428],[63,18444,18446],{"className":18445},[433,434,435,436],[63,18447,612],{"className":18448},[399,472,436],[63,18450,18451,18454],{"style":16572},[63,18452],{"className":18453,"style":817},[428],[63,18455,18457],{"className":18456},[433,434,435,436],[63,18458,1057],{"className":18459},[399,472,436],[63,18461,507],{"className":18462},[506],[63,18464,18466],{"className":18465},[416],[63,18467,18469],{"className":18468,"style":17487},[420],[63,18470],{},[63,18472,1092],{"className":18473},[593],[63,18475,507],{"className":18476},[506],[63,18478,18480],{"className":18479},[416],[63,18481,18483],{"className":18482,"style":15516},[420],[63,18484],{},[14,18486,18487],{},[63,18488,18490,18597],{"className":18489},[342],[63,18491,18493],{"className":18492},[346],[348,18494,18495],{"xmlns":350},[352,18496,18497,18594],{},[355,18498,18499,18501,18503,18505,18507,18509,18512,18518,18532,18550,18552,18554,18560,18562,18564,18566,18572,18574,18582,18584,18592],{},[371,18500,7842],{},[371,18502,1072],{},[371,18504,12890],{},[371,18506,12890],{},[738,18508,204],{},[738,18510,18511],{},"−",[358,18513,18514,18516],{},[361,18515,363],{},[371,18517,7342],{},[1870,18519,18520,18522,18530],{},[738,18521,1874],{},[355,18523,18524,18526,18528],{},[371,18525,612],{},[738,18527,204],{},[361,18529,363],{},[371,18531,7342],{},[1870,18533,18534,18536,18544],{},[738,18535,1874],{},[355,18537,18538,18540,18542],{},[371,18539,1057],{},[738,18541,204],{},[361,18543,363],{},[368,18545,18546,18548],{},[371,18547,13358],{},[371,18549,612],{},[371,18551,747],{},[738,18553,1078],{"stretchy":1077},[742,18555,18556,18558],{},[371,18557,13274],{},[371,18559,612],{},[738,18561,1092],{"stretchy":1077},[371,18563,14950],{},[738,18565,1100],{},[368,18567,18568,18570],{},[371,18569,5685],{},[371,18571,14101],{},[738,18573,1078],{"stretchy":1077},[1870,18575,18576,18578,18580],{},[371,18577,38],{},[371,18579,612],{},[371,18581,1057],{},[371,18583,13030],{"mathvariant":1053},[1870,18585,18586,18588,18590],{},[371,18587,12890],{},[371,18589,612],{},[371,18591,1057],{},[738,18593,1092],{"stretchy":1077},[378,18595,18596],{"encoding":380},"Loss = -\\frac{1}{N}\\sum\\limits_{n=1}^N \\sum\\limits_{t=1}^{T_n} R(\\tau^n) \\log P_\\theta(a_n^t|s_n^t)",[63,18598,18600,18623],{"className":18599,"ariaHidden":386},[385],[63,18601,18603,18607,18610,18614,18617,18620],{"className":18602},[390],[63,18604],{"className":18605,"style":18606},[394],"height:0.6833em;",[63,18608,7842],{"className":18609},[399,472],[63,18611,18613],{"className":18612},[399,472],"oss",[63,18615],{"className":18616,"style":777},[776],[63,18618,204],{"className":18619},[781],[63,18621],{"className":18622,"style":777},[776],[63,18624,18626,18630,18633,18703,18706,18774,18777,18883,18886,18889,18892,18921,18924,18927,18932,18935,18975,18978,19030,19033,19084],{"className":18625},[390],[63,18627],{"className":18628,"style":18629},[394],"height:2.5065em;vertical-align:-0.9671em;",[63,18631,18511],{"className":18632},[399],[63,18634,18636,18639,18700],{"className":18635},[399],[63,18637],{"className":18638},[403,404],[63,18640,18642],{"className":18641},[358],[63,18643,18645,18691],{"className":18644},[411,412],[63,18646,18648,18688],{"className":18647},[416],[63,18649,18651,18666,18674],{"className":18650,"style":421},[420],[63,18652,18654,18657],{"style":18653},"top:-2.655em;",[63,18655],{"className":18656,"style":429},[428],[63,18658,18660],{"className":18659},[433,434,435,436],[63,18661,18663],{"className":18662},[399,436],[63,18664,7342],{"className":18665,"style":16436},[399,472,436],[63,18667,18668,18671],{"style":554},[63,18669],{"className":18670,"style":429},[428],[63,18672],{"className":18673,"style":562},[561],[63,18675,18676,18679],{"style":565},[63,18677],{"className":18678,"style":429},[428],[63,18680,18682],{"className":18681},[433,434,435,436],[63,18683,18685],{"className":18684},[399,436],[63,18686,363],{"className":18687},[399,436],[63,18689,507],{"className":18690},[506],[63,18692,18694],{"className":18693},[416],[63,18695,18698],{"className":18696,"style":18697},[420],"height:0.345em;",[63,18699],{},[63,18701],{"className":18702},[593,404],[63,18704],{"className":18705,"style":1177},[776],[63,18707,18709],{"className":18708},[1212,14383],[63,18710,18712,18765],{"className":18711},[411,412],[63,18713,18715,18762],{"className":18714},[416],[63,18716,18719,18740,18750],{"className":18717,"style":18718},[420],"height:1.5283em;",[63,18720,18722,18725],{"style":18721},"top:-2.1329em;margin-left:0em;",[63,18723],{"className":18724,"style":429},[428],[63,18726,18728],{"className":18727},[433,434,435,436],[63,18729,18731,18734,18737],{"className":18730},[399,436],[63,18732,612],{"className":18733},[399,472,436],[63,18735,204],{"className":18736},[781,436],[63,18738,363],{"className":18739},[399,436],[63,18741,18742,18745],{"style":458},[63,18743],{"className":18744,"style":429},[428],[63,18746,18747],{},[63,18748,1874],{"className":18749},[1212,1958,1959],[63,18751,18753,18756],{"style":18752},"top:-3.95em;margin-left:0em;",[63,18754],{"className":18755,"style":429},[428],[63,18757,18759],{"className":18758},[433,434,435,436],[63,18760,7342],{"className":18761,"style":16436},[399,472,436],[63,18763,507],{"className":18764},[506],[63,18766,18768],{"className":18767},[416],[63,18769,18772],{"className":18770,"style":18771},[420],"height:0.9671em;",[63,18773],{},[63,18775],{"className":18776,"style":1177},[776],[63,18778,18780],{"className":18779},[1212,14383],[63,18781,18783,18875],{"className":18782},[411,412],[63,18784,18786,18872],{"className":18785},[416],[63,18787,18790,18810,18820],{"className":18788,"style":18789},[420],"height:1.5394em;",[63,18791,18792,18795],{"style":18721},[63,18793],{"className":18794,"style":429},[428],[63,18796,18798],{"className":18797},[433,434,435,436],[63,18799,18801,18804,18807],{"className":18800},[399,436],[63,18802,1057],{"className":18803},[399,472,436],[63,18805,204],{"className":18806},[781,436],[63,18808,363],{"className":18809},[399,436],[63,18811,18812,18815],{"style":458},[63,18813],{"className":18814,"style":429},[428],[63,18816,18817],{},[63,18818,1874],{"className":18819},[1212,1958,1959],[63,18821,18823,18826],{"style":18822},"top:-3.9611em;margin-left:0em;",[63,18824],{"className":18825,"style":429},[428],[63,18827,18829],{"className":18828},[433,434,435,436],[63,18830,18832],{"className":18831},[399,436],[63,18833,18835,18838],{"className":18834},[399,436],[63,18836,13358],{"className":18837,"style":3251},[399,472,436],[63,18839,18841],{"className":18840},[476],[63,18842,18844,18864],{"className":18843},[411,412],[63,18845,18847,18861],{"className":18846},[416],[63,18848,18850],{"className":18849,"style":4367},[420],[63,18851,18852,18855],{"style":17358},[63,18853],{"className":18854,"style":493},[428],[63,18856,18858],{"className":18857},[433,497,498,436],[63,18859,612],{"className":18860},[399,472,436],[63,18862,507],{"className":18863},[506],[63,18865,18867],{"className":18866},[416],[63,18868,18870],{"className":18869,"style":4389},[420],[63,18871],{},[63,18873,507],{"className":18874},[506],[63,18876,18878],{"className":18877},[416],[63,18879,18881],{"className":18880,"style":18771},[420],[63,18882],{},[63,18884],{"className":18885,"style":1177},[776],[63,18887,747],{"className":18888,"style":14254},[399,472],[63,18890,1078],{"className":18891},[403],[63,18893,18895,18898],{"className":18894},[399],[63,18896,13274],{"className":18897,"style":13290},[399,472],[63,18899,18901],{"className":18900},[476],[63,18902,18904],{"className":18903},[411],[63,18905,18907],{"className":18906},[416],[63,18908,18910],{"className":18909,"style":16768},[420],[63,18911,18912,18915],{"style":813},[63,18913],{"className":18914,"style":817},[428],[63,18916,18918],{"className":18917},[433,434,435,436],[63,18919,612],{"className":18920},[399,472,436],[63,18922,1092],{"className":18923},[593],[63,18925],{"className":18926,"style":1177},[776],[63,18928,16996,18930],{"className":18929},[1212],[63,18931,17000],{"style":16999},[63,18933],{"className":18934,"style":1177},[776],[63,18936,18938,18941],{"className":18937},[399],[63,18939,5685],{"className":18940,"style":3251},[399,472],[63,18942,18944],{"className":18943},[476],[63,18945,18947,18967],{"className":18946},[411,412],[63,18948,18950,18964],{"className":18949},[416],[63,18951,18953],{"className":18952,"style":1532},[420],[63,18954,18955,18958],{"style":7483},[63,18956],{"className":18957,"style":817},[428],[63,18959,18961],{"className":18960},[433,434,435,436],[63,18962,14101],{"className":18963,"style":3276},[399,472,436],[63,18965,507],{"className":18966},[506],[63,18968,18970],{"className":18969},[416],[63,18971,18973],{"className":18972,"style":1554},[420],[63,18974],{},[63,18976,1078],{"className":18977},[403],[63,18979,18981,18984],{"className":18980},[399],[63,18982,38],{"className":18983},[399,472],[63,18985,18987],{"className":18986},[476],[63,18988,18990,19022],{"className":18989},[411,412],[63,18991,18993,19019],{"className":18992},[416],[63,18994,18997,19008],{"className":18995,"style":18996},[420],"height:0.7936em;",[63,18998,18999,19002],{"style":17457},[63,19000],{"className":19001,"style":817},[428],[63,19003,19005],{"className":19004},[433,434,435,436],[63,19006,612],{"className":19007},[399,472,436],[63,19009,19010,19013],{"style":813},[63,19011],{"className":19012,"style":817},[428],[63,19014,19016],{"className":19015},[433,434,435,436],[63,19017,1057],{"className":19018},[399,472,436],[63,19020,507],{"className":19021},[506],[63,19023,19025],{"className":19024},[416],[63,19026,19028],{"className":19027,"style":17487},[420],[63,19029],{},[63,19031,13030],{"className":19032},[399],[63,19034,19036,19039],{"className":19035},[399],[63,19037,12890],{"className":19038},[399,472],[63,19040,19042],{"className":19041},[476],[63,19043,19045,19076],{"className":19044},[411,412],[63,19046,19048,19073],{"className":19047},[416],[63,19049,19051,19062],{"className":19050,"style":18996},[420],[63,19052,19053,19056],{"style":17457},[63,19054],{"className":19055,"style":817},[428],[63,19057,19059],{"className":19058},[433,434,435,436],[63,19060,612],{"className":19061},[399,472,436],[63,19063,19064,19067],{"style":813},[63,19065],{"className":19066,"style":817},[428],[63,19068,19070],{"className":19069},[433,434,435,436],[63,19071,1057],{"className":19072},[399,472,436],[63,19074,507],{"className":19075},[506],[63,19077,19079],{"className":19078},[416],[63,19080,19082],{"className":19081,"style":17487},[420],[63,19083],{},[63,19085,1092],{"className":19086},[593],[14,19088,19089],{},[63,19090,19092,19180],{"className":19091},[342],[63,19093,19095],{"className":19094},[346],[348,19096,19097],{"xmlns":350},[352,19098,19099,19177],{},[355,19100,19101,19103,19105,19111,19113,19115,19138,19155,19167,19169],{},[371,19102,747],{},[738,19104,1078],{"stretchy":1077},[742,19106,19107,19109],{},[371,19108,13274],{},[371,19110,612],{},[738,19112,1092],{"stretchy":1077},[738,19114,204],{},[1870,19116,19117,19119,19132],{},[738,19118,1874],{},[355,19120,19121,19123,19125],{},[371,19122,1057],{},[738,19124,204],{},[742,19126,19127,19129],{},[371,19128,1057],{},[738,19130,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},"′",[368,19133,19134,19136],{},[371,19135,13358],{},[371,19137,612],{},[742,19139,19140,19143],{},[371,19141,19142],{},"γ",[355,19144,19145,19147,19149],{},[371,19146,1057],{},[738,19148,18511],{},[742,19150,19151,19153],{},[371,19152,1057],{},[738,19154,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[1870,19156,19157,19159,19165],{},[371,19158,7852],{},[742,19160,19161,19163],{},[371,19162,1057],{},[738,19164,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[371,19166,612],{},[738,19168,204],{},[1870,19170,19171,19173,19175],{},[371,19172,747],{},[371,19174,1057],{},[371,19176,612],{},[378,19178,19179],{"encoding":380},"R(\\tau^n) = \\sum\\limits_{t=t'}^{T_n} \\gamma^{t-t'} r_{t'}^n = R_t^n",[63,19181,19183,19236,19547],{"className":19182,"ariaHidden":386},[385],[63,19184,19186,19189,19192,19195,19224,19227,19230,19233],{"className":19185},[390],[63,19187],{"className":19188,"style":1155},[394],[63,19190,747],{"className":19191,"style":14254},[399,472],[63,19193,1078],{"className":19194},[403],[63,19196,19198,19201],{"className":19197},[399],[63,19199,13274],{"className":19200,"style":13290},[399,472],[63,19202,19204],{"className":19203},[476],[63,19205,19207],{"className":19206},[411],[63,19208,19210],{"className":19209},[416],[63,19211,19213],{"className":19212,"style":16768},[420],[63,19214,19215,19218],{"style":813},[63,19216],{"className":19217,"style":817},[428],[63,19219,19221],{"className":19220},[433,434,435,436],[63,19222,612],{"className":19223},[399,472,436],[63,19225,1092],{"className":19226},[593],[63,19228],{"className":19229,"style":777},[776],[63,19231,204],{"className":19232},[781],[63,19234],{"className":19235,"style":777},[776],[63,19237,19239,19243,19380,19383,19453,19538,19541,19544],{"className":19238},[390],[63,19240],{"className":19241,"style":19242},[394],"height:2.5334em;vertical-align:-0.994em;",[63,19244,19246],{"className":19245},[1212,14383],[63,19247,19249,19371],{"className":19248},[411,412],[63,19250,19252,19368],{"className":19251},[416],[63,19253,19255,19307,19317],{"className":19254,"style":18789},[420],[63,19256,19258,19261],{"style":19257},"top:-2.106em;margin-left:0em;",[63,19259],{"className":19260,"style":429},[428],[63,19262,19264],{"className":19263},[433,434,435,436],[63,19265,19267,19270,19273],{"className":19266},[399,436],[63,19268,1057],{"className":19269},[399,472,436],[63,19271,204],{"className":19272},[781,436],[63,19274,19276,19279],{"className":19275},[399,436],[63,19277,1057],{"className":19278},[399,472,436],[63,19280,19282],{"className":19281},[476],[63,19283,19285],{"className":19284},[411],[63,19286,19288],{"className":19287},[416],[63,19289,19292],{"className":19290,"style":19291},[420],"height:0.6828em;",[63,19293,19295,19298],{"style":19294},"top:-2.786em;margin-right:0.0714em;",[63,19296],{"className":19297,"style":493},[428],[63,19299,19301],{"className":19300},[433,497,498,436],[63,19302,19304],{"className":19303},[399,436],[63,19305,19131],{"className":19306},[399,436],[63,19308,19309,19312],{"style":458},[63,19310],{"className":19311,"style":429},[428],[63,19313,19314],{},[63,19315,1874],{"className":19316},[1212,1958,1959],[63,19318,19319,19322],{"style":18822},[63,19320],{"className":19321,"style":429},[428],[63,19323,19325],{"className":19324},[433,434,435,436],[63,19326,19328],{"className":19327},[399,436],[63,19329,19331,19334],{"className":19330},[399,436],[63,19332,13358],{"className":19333,"style":3251},[399,472,436],[63,19335,19337],{"className":19336},[476],[63,19338,19340,19360],{"className":19339},[411,412],[63,19341,19343,19357],{"className":19342},[416],[63,19344,19346],{"className":19345,"style":4367},[420],[63,19347,19348,19351],{"style":17358},[63,19349],{"className":19350,"style":493},[428],[63,19352,19354],{"className":19353},[433,497,498,436],[63,19355,612],{"className":19356},[399,472,436],[63,19358,507],{"className":19359},[506],[63,19361,19363],{"className":19362},[416],[63,19364,19366],{"className":19365,"style":4389},[420],[63,19367],{},[63,19369,507],{"className":19370},[506],[63,19372,19374],{"className":19373},[416],[63,19375,19378],{"className":19376,"style":19377},[420],"height:0.994em;",[63,19379],{},[63,19381],{"className":19382,"style":1177},[776],[63,19384,19386,19390],{"className":19385},[399],[63,19387,19142],{"className":19388,"style":19389},[399,472],"margin-right:0.05556em;",[63,19391,19393],{"className":19392},[476],[63,19394,19396],{"className":19395},[411],[63,19397,19399],{"className":19398},[416],[63,19400,19403],{"className":19401,"style":19402},[420],"height:0.9425em;",[63,19404,19405,19408],{"style":813},[63,19406],{"className":19407,"style":817},[428],[63,19409,19411],{"className":19410},[433,434,435,436],[63,19412,19414,19417,19420],{"className":19413},[399,436],[63,19415,1057],{"className":19416},[399,472,436],[63,19418,18511],{"className":19419},[830,436],[63,19421,19423,19426],{"className":19422},[399,436],[63,19424,1057],{"className":19425},[399,472,436],[63,19427,19429],{"className":19428},[476],[63,19430,19432],{"className":19431},[411],[63,19433,19435],{"className":19434},[416],[63,19436,19439],{"className":19437,"style":19438},[420],"height:0.8278em;",[63,19440,19441,19444],{"style":1358},[63,19442],{"className":19443,"style":493},[428],[63,19445,19447],{"className":19446},[433,497,498,436],[63,19448,19450],{"className":19449},[399,436],[63,19451,19131],{"className":19452},[399,436],[63,19454,19456,19459],{"className":19455},[399],[63,19457,7852],{"className":19458,"style":3276},[399,472],[63,19460,19462],{"className":19461},[476],[63,19463,19465,19529],{"className":19464},[411,412],[63,19466,19468,19526],{"className":19467},[416],[63,19469,19471,19515],{"className":19470,"style":16768},[420],[63,19472,19474,19477],{"style":19473},"top:-2.425em;margin-left:-0.0278em;margin-right:0.05em;",[63,19475],{"className":19476,"style":817},[428],[63,19478,19480],{"className":19479},[433,434,435,436],[63,19481,19483],{"className":19482},[399,436],[63,19484,19486,19489],{"className":19485},[399,436],[63,19487,1057],{"className":19488},[399,472,436],[63,19490,19492],{"className":19491},[476],[63,19493,19495],{"className":19494},[411],[63,19496,19498],{"className":19497},[416],[63,19499,19501],{"className":19500,"style":19291},[420],[63,19502,19503,19506],{"style":19294},[63,19504],{"className":19505,"style":493},[428],[63,19507,19509],{"className":19508},[433,497,498,436],[63,19510,19512],{"className":19511},[399,436],[63,19513,19131],{"className":19514},[399,436],[63,19516,19517,19520],{"style":813},[63,19518],{"className":19519,"style":817},[428],[63,19521,19523],{"className":19522},[433,434,435,436],[63,19524,612],{"className":19525},[399,472,436],[63,19527,507],{"className":19528},[506],[63,19530,19532],{"className":19531},[416],[63,19533,19536],{"className":19534,"style":19535},[420],"height:0.275em;",[63,19537],{},[63,19539],{"className":19540,"style":777},[776],[63,19542,204],{"className":19543},[781],[63,19545],{"className":19546,"style":777},[776],[63,19548,19550,19554],{"className":19549},[390],[63,19551],{"className":19552,"style":19553},[394],"height:0.9303em;vertical-align:-0.247em;",[63,19555,19557,19560],{"className":19556},[399],[63,19558,747],{"className":19559,"style":14254},[399,472],[63,19561,19563],{"className":19562},[476],[63,19564,19566,19598],{"className":19565},[411,412],[63,19567,19569,19595],{"className":19568},[416],[63,19570,19572,19584],{"className":19571,"style":16768},[420],[63,19573,19575,19578],{"style":19574},"top:-2.453em;margin-left:-0.0077em;margin-right:0.05em;",[63,19576],{"className":19577,"style":817},[428],[63,19579,19581],{"className":19580},[433,434,435,436],[63,19582,1057],{"className":19583},[399,472,436],[63,19585,19586,19589],{"style":813},[63,19587],{"className":19588,"style":817},[428],[63,19590,19592],{"className":19591},[433,434,435,436],[63,19593,612],{"className":19594},[399,472,436],[63,19596,507],{"className":19597},[506],[63,19599,19601],{"className":19600},[416],[63,19602,19604],{"className":19603,"style":17487},[420],[63,19605],{},[10,19607,19609],{"id":19608},"actor-critic-演员-评论家算法","Actor-Critic 演员-评论家算法",[14,19611,19612,19613,19684,19685,19772],{},"为了衡量某个动作\"相对好坏\"，我们让奖励 ",[63,19614,19616,19634],{"className":19615},[342],[63,19617,19619],{"className":19618},[346],[348,19620,19621],{"xmlns":350},[352,19622,19623,19631],{},[355,19624,19625],{},[368,19626,19627,19629],{},[371,19628,747],{},[371,19630,1057],{},[378,19632,19633],{"encoding":380},"R_t",[63,19635,19637],{"className":19636,"ariaHidden":386},[385],[63,19638,19640,19643],{"className":19639},[390],[63,19641],{"className":19642,"style":6995},[394],[63,19644,19646,19649],{"className":19645},[399],[63,19647,747],{"className":19648,"style":14254},[399,472],[63,19650,19652],{"className":19651},[476],[63,19653,19655,19676],{"className":19654},[411,412],[63,19656,19658,19673],{"className":19657},[416],[63,19659,19661],{"className":19660,"style":12818},[420],[63,19662,19664,19667],{"style":19663},"top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;",[63,19665],{"className":19666,"style":817},[428],[63,19668,19670],{"className":19669},[433,434,435,436],[63,19671,1057],{"className":19672},[399,472,436],[63,19674,507],{"className":19675},[506],[63,19677,19679],{"className":19678},[416],[63,19680,19682],{"className":19681,"style":1554},[420],[63,19683],{}," 减去一个基准值 ",[63,19686,19688,19713],{"className":19687},[342],[63,19689,19691],{"className":19690},[346],[348,19692,19693],{"xmlns":350},[352,19694,19695,19710],{},[355,19696,19697,19700,19702,19708],{},[371,19698,19699],{},"B",[738,19701,1078],{"stretchy":1077},[368,19703,19704,19706],{},[371,19705,12890],{},[371,19707,1057],{},[738,19709,1092],{"stretchy":1077},[378,19711,19712],{"encoding":380},"B(s_t)",[63,19714,19716],{"className":19715,"ariaHidden":386},[385],[63,19717,19719,19722,19726,19729,19769],{"className":19718},[390],[63,19720],{"className":19721,"style":1155},[394],[63,19723,19699],{"className":19724,"style":19725},[399,472],"margin-right:0.05017em;",[63,19727,1078],{"className":19728},[403],[63,19730,19732,19735],{"className":19731},[399],[63,19733,12890],{"className":19734},[399,472],[63,19736,19738],{"className":19737},[476],[63,19739,19741,19761],{"className":19740},[411,412],[63,19742,19744,19758],{"className":19743},[416],[63,19745,19747],{"className":19746,"style":12818},[420],[63,19748,19749,19752],{"style":1535},[63,19750],{"className":19751,"style":817},[428],[63,19753,19755],{"className":19754},[433,434,435,436],[63,19756,1057],{"className":19757},[399,472,436],[63,19759,507],{"className":19760},[506],[63,19762,19764],{"className":19763},[416],[63,19765,19767],{"className":19766,"style":1554},[420],[63,19768],{},[63,19770,1092],{"className":19771},[593],"，从而降低梯度估计的方差：",[14,19774,19775],{},[63,19776,19778,19890],{"className":19777},[342],[63,19779,19781],{"className":19780},[346],[348,19782,19783],{"xmlns":350},[352,19784,19785,19887],{},[355,19786,19787,19793,19807,19825,19827,19835,19837,19839,19841,19849,19851,19853,19855,19857,19859,19865,19867,19875,19877,19885],{},[358,19788,19789,19791],{},[361,19790,363],{},[371,19792,7342],{},[1870,19794,19795,19797,19805],{},[738,19796,1874],{},[355,19798,19799,19801,19803],{},[371,19800,612],{},[738,19802,204],{},[361,19804,363],{},[371,19806,7342],{},[1870,19808,19809,19811,19819],{},[738,19810,1874],{},[355,19812,19813,19815,19817],{},[371,19814,1057],{},[738,19816,204],{},[361,19818,363],{},[368,19820,19821,19823],{},[371,19822,13358],{},[371,19824,612],{},[738,19826,1078],{"stretchy":1077},[1870,19828,19829,19831,19833],{},[371,19830,747],{},[371,19832,1057],{},[371,19834,612],{},[738,19836,18511],{},[371,19838,19699],{},[738,19840,1078],{"stretchy":1077},[1870,19842,19843,19845,19847],{},[371,19844,12890],{},[371,19846,1057],{},[371,19848,612],{},[738,19850,1092],{"stretchy":1077},[738,19852,1092],{"stretchy":1077},[371,19854,14525],{"mathvariant":1053},[371,19856,14950],{},[738,19858,1100],{},[368,19860,19861,19863],{},[371,19862,5685],{},[371,19864,14101],{},[738,19866,1078],{"stretchy":1077},[1870,19868,19869,19871,19873],{},[371,19870,38],{},[371,19872,1057],{},[371,19874,612],{},[371,19876,13030],{"mathvariant":1053},[1870,19878,19879,19881,19883],{},[371,19880,12890],{},[371,19882,1057],{},[371,19884,612],{},[738,19886,1092],{"stretchy":1077},[378,19888,19889],{"encoding":380}," \\frac{1}{N} \\sum\\limits_{n=1}^N \\sum\\limits_{t=1}^{T_n} (R_t^n - B(s_t^n))\\nabla \\log P_\\theta(a_t^n|s_t^n)",[63,19891,19893,20204],{"className":19892,"ariaHidden":386},[385],[63,19894,19896,19899,19967,19970,20034,20037,20141,20144,20195,20198,20201],{"className":19895},[390],[63,19897],{"className":19898,"style":18629},[394],[63,19900,19902,19905,19964],{"className":19901},[399],[63,19903],{"className":19904},[403,404],[63,19906,19908],{"className":19907},[358],[63,19909,19911,19956],{"className":19910},[411,412],[63,19912,19914,19953],{"className":19913},[416],[63,19915,19917,19931,19939],{"className":19916,"style":421},[420],[63,19918,19919,19922],{"style":18653},[63,19920],{"className":19921,"style":429},[428],[63,19923,19925],{"className":19924},[433,434,435,436],[63,19926,19928],{"className":19927},[399,436],[63,19929,7342],{"className":19930,"style":16436},[399,472,436],[63,19932,19933,19936],{"style":554},[63,19934],{"className":19935,"style":429},[428],[63,19937],{"className":19938,"style":562},[561],[63,19940,19941,19944],{"style":565},[63,19942],{"className":19943,"style":429},[428],[63,19945,19947],{"className":19946},[433,434,435,436],[63,19948,19950],{"className":19949},[399,436],[63,19951,363],{"className":19952},[399,436],[63,19954,507],{"className":19955},[506],[63,19957,19959],{"className":19958},[416],[63,19960,19962],{"className":19961,"style":18697},[420],[63,19963],{},[63,19965],{"className":19966},[593,404],[63,19968],{"className":19969,"style":1177},[776],[63,19971,19973],{"className":19972},[1212,14383],[63,19974,19976,20026],{"className":19975},[411,412],[63,19977,19979,20023],{"className":19978},[416],[63,19980,19982,20002,20012],{"className":19981,"style":18718},[420],[63,19983,19984,19987],{"style":18721},[63,19985],{"className":19986,"style":429},[428],[63,19988,19990],{"className":19989},[433,434,435,436],[63,19991,19993,19996,19999],{"className":19992},[399,436],[63,19994,612],{"className":19995},[399,472,436],[63,19997,204],{"className":19998},[781,436],[63,20000,363],{"className":20001},[399,436],[63,20003,20004,20007],{"style":458},[63,20005],{"className":20006,"style":429},[428],[63,20008,20009],{},[63,20010,1874],{"className":20011},[1212,1958,1959],[63,20013,20014,20017],{"style":18752},[63,20015],{"className":20016,"style":429},[428],[63,20018,20020],{"className":20019},[433,434,435,436],[63,20021,7342],{"className":20022,"style":16436},[399,472,436],[63,20024,507],{"className":20025},[506],[63,20027,20029],{"className":20028},[416],[63,20030,20032],{"className":20031,"style":18771},[420],[63,20033],{},[63,20035],{"className":20036,"style":1177},[776],[63,20038,20040],{"className":20039},[1212,14383],[63,20041,20043,20133],{"className":20042},[411,412],[63,20044,20046,20130],{"className":20045},[416],[63,20047,20049,20069,20079],{"className":20048,"style":18789},[420],[63,20050,20051,20054],{"style":18721},[63,20052],{"className":20053,"style":429},[428],[63,20055,20057],{"className":20056},[433,434,435,436],[63,20058,20060,20063,20066],{"className":20059},[399,436],[63,20061,1057],{"className":20062},[399,472,436],[63,20064,204],{"className":20065},[781,436],[63,20067,363],{"className":20068},[399,436],[63,20070,20071,20074],{"style":458},[63,20072],{"className":20073,"style":429},[428],[63,20075,20076],{},[63,20077,1874],{"className":20078},[1212,1958,1959],[63,20080,20081,20084],{"style":18822},[63,20082],{"className":20083,"style":429},[428],[63,20085,20087],{"className":20086},[433,434,435,436],[63,20088,20090],{"className":20089},[399,436],[63,20091,20093,20096],{"className":20092},[399,436],[63,20094,13358],{"className":20095,"style":3251},[399,472,436],[63,20097,20099],{"className":20098},[476],[63,20100,20102,20122],{"className":20101},[411,412],[63,20103,20105,20119],{"className":20104},[416],[63,20106,20108],{"className":20107,"style":4367},[420],[63,20109,20110,20113],{"style":17358},[63,20111],{"className":20112,"style":493},[428],[63,20114,20116],{"className":20115},[433,497,498,436],[63,20117,612],{"className":20118},[399,472,436],[63,20120,507],{"className":20121},[506],[63,20123,20125],{"className":20124},[416],[63,20126,20128],{"className":20127,"style":4389},[420],[63,20129],{},[63,20131,507],{"className":20132},[506],[63,20134,20136],{"className":20135},[416],[63,20137,20139],{"className":20138,"style":18771},[420],[63,20140],{},[63,20142,1078],{"className":20143},[403],[63,20145,20147,20150],{"className":20146},[399],[63,20148,747],{"className":20149,"style":14254},[399,472],[63,20151,20153],{"className":20152},[476],[63,20154,20156,20187],{"className":20155},[411,412],[63,20157,20159,20184],{"className":20158},[416],[63,20160,20162,20173],{"className":20161,"style":16768},[420],[63,20163,20164,20167],{"style":19574},[63,20165],{"className":20166,"style":817},[428],[63,20168,20170],{"className":20169},[433,434,435,436],[63,20171,1057],{"className":20172},[399,472,436],[63,20174,20175,20178],{"style":813},[63,20176],{"className":20177,"style":817},[428],[63,20179,20181],{"className":20180},[433,434,435,436],[63,20182,612],{"className":20183},[399,472,436],[63,20185,507],{"className":20186},[506],[63,20188,20190],{"className":20189},[416],[63,20191,20193],{"className":20192,"style":17487},[420],[63,20194],{},[63,20196],{"className":20197,"style":1920},[776],[63,20199,18511],{"className":20200},[830],[63,20202],{"className":20203,"style":1920},[776],[63,20205,20207,20210,20213,20216,20267,20270,20273,20276,20281,20284,20324,20327,20378,20381,20432],{"className":20206},[390],[63,20208],{"className":20209,"style":1155},[394],[63,20211,19699],{"className":20212,"style":19725},[399,472],[63,20214,1078],{"className":20215},[403],[63,20217,20219,20222],{"className":20218},[399],[63,20220,12890],{"className":20221},[399,472],[63,20223,20225],{"className":20224},[476],[63,20226,20228,20259],{"className":20227},[411,412],[63,20229,20231,20256],{"className":20230},[416],[63,20232,20234,20245],{"className":20233,"style":16768},[420],[63,20235,20236,20239],{"style":17457},[63,20237],{"className":20238,"style":817},[428],[63,20240,20242],{"className":20241},[433,434,435,436],[63,20243,1057],{"className":20244},[399,472,436],[63,20246,20247,20250],{"style":813},[63,20248],{"className":20249,"style":817},[428],[63,20251,20253],{"className":20252},[433,434,435,436],[63,20254,612],{"className":20255},[399,472,436],[63,20257,507],{"className":20258},[506],[63,20260,20262],{"className":20261},[416],[63,20263,20265],{"className":20264,"style":17487},[420],[63,20266],{},[63,20268,7950],{"className":20269},[593],[63,20271,14525],{"className":20272},[399],[63,20274],{"className":20275,"style":1177},[776],[63,20277,16996,20279],{"className":20278},[1212],[63,20280,17000],{"style":16999},[63,20282],{"className":20283,"style":1177},[776],[63,20285,20287,20290],{"className":20286},[399],[63,20288,5685],{"className":20289,"style":3251},[399,472],[63,20291,20293],{"className":20292},[476],[63,20294,20296,20316],{"className":20295},[411,412],[63,20297,20299,20313],{"className":20298},[416],[63,20300,20302],{"className":20301,"style":1532},[420],[63,20303,20304,20307],{"style":7483},[63,20305],{"className":20306,"style":817},[428],[63,20308,20310],{"className":20309},[433,434,435,436],[63,20311,14101],{"className":20312,"style":3276},[399,472,436],[63,20314,507],{"className":20315},[506],[63,20317,20319],{"className":20318},[416],[63,20320,20322],{"className":20321,"style":1554},[420],[63,20323],{},[63,20325,1078],{"className":20326},[403],[63,20328,20330,20333],{"className":20329},[399],[63,20331,38],{"className":20332},[399,472],[63,20334,20336],{"className":20335},[476],[63,20337,20339,20370],{"className":20338},[411,412],[63,20340,20342,20367],{"className":20341},[416],[63,20343,20345,20356],{"className":20344,"style":16768},[420],[63,20346,20347,20350],{"style":17457},[63,20348],{"className":20349,"style":817},[428],[63,20351,20353],{"className":20352},[433,434,435,436],[63,20354,1057],{"className":20355},[399,472,436],[63,20357,20358,20361],{"style":813},[63,20359],{"className":20360,"style":817},[428],[63,20362,20364],{"className":20363},[433,434,435,436],[63,20365,612],{"className":20366},[399,472,436],[63,20368,507],{"className":20369},[506],[63,20371,20373],{"className":20372},[416],[63,20374,20376],{"className":20375,"style":17487},[420],[63,20377],{},[63,20379,13030],{"className":20380},[399],[63,20382,20384,20387],{"className":20383},[399],[63,20385,12890],{"className":20386},[399,472],[63,20388,20390],{"className":20389},[476],[63,20391,20393,20424],{"className":20392},[411,412],[63,20394,20396,20421],{"className":20395},[416],[63,20397,20399,20410],{"className":20398,"style":16768},[420],[63,20400,20401,20404],{"style":17457},[63,20402],{"className":20403,"style":817},[428],[63,20405,20407],{"className":20406},[433,434,435,436],[63,20408,1057],{"className":20409},[399,472,436],[63,20411,20412,20415],{"style":813},[63,20413],{"className":20414,"style":817},[428],[63,20416,20418],{"className":20417},[433,434,435,436],[63,20419,612],{"className":20420},[399,472,436],[63,20422,507],{"className":20423},[506],[63,20425,20427],{"className":20426},[416],[63,20428,20430],{"className":20429,"style":17487},[420],[63,20431],{},[63,20433,1092],{"className":20434},[593],[14,20436,20437,20438,20521,20522,20620,20621,20649,20650,20678,20679,20682,20683,20620,20769,20797,20798,235],{},"然而 ",[63,20439,20441,20461],{"className":20440},[342],[63,20442,20444],{"className":20443},[346],[348,20445,20446],{"xmlns":350},[352,20447,20448,20458],{},[355,20449,20450],{},[1870,20451,20452,20454,20456],{},[371,20453,747],{},[371,20455,1057],{},[371,20457,612],{},[378,20459,20460],{"encoding":380},"R_t^n",[63,20462,20464],{"className":20463,"ariaHidden":386},[385],[63,20465,20467,20470],{"className":20466},[390],[63,20468],{"className":20469,"style":19553},[394],[63,20471,20473,20476],{"className":20472},[399],[63,20474,747],{"className":20475,"style":14254},[399,472],[63,20477,20479],{"className":20478},[476],[63,20480,20482,20513],{"className":20481},[411,412],[63,20483,20485,20510],{"className":20484},[416],[63,20486,20488,20499],{"className":20487,"style":16768},[420],[63,20489,20490,20493],{"style":19574},[63,20491],{"className":20492,"style":817},[428],[63,20494,20496],{"className":20495},[433,434,435,436],[63,20497,1057],{"className":20498},[399,472,436],[63,20500,20501,20504],{"style":813},[63,20502],{"className":20503,"style":817},[428],[63,20505,20507],{"className":20506},[433,434,435,436],[63,20508,612],{"className":20509},[399,472,436],[63,20511,507],{"className":20512},[506],[63,20514,20516],{"className":20515},[416],[63,20517,20519],{"className":20518,"style":17487},[420],[63,20520],{}," 仅来自一次随机采样，方差很大、训练不稳定。一个自然的改进是引入价值函数：用 ",[63,20523,20525,20553],{"className":20524},[342],[63,20526,20528],{"className":20527},[346],[348,20529,20530],{"xmlns":350},[352,20531,20532,20550],{},[355,20533,20534,20540,20542,20544,20546,20548],{},[368,20535,20536,20538],{},[371,20537,736],{},[371,20539,14101],{},[738,20541,1078],{"stretchy":1077},[371,20543,12890],{},[738,20545,1083],{"separator":386},[371,20547,38],{},[738,20549,1092],{"stretchy":1077},[378,20551,20552],{"encoding":380},"Q_\\theta(s,a)",[63,20554,20556],{"className":20555,"ariaHidden":386},[385],[63,20557,20559,20562,20602,20605,20608,20611,20614,20617],{"className":20558},[390],[63,20560],{"className":20561,"style":1155},[394],[63,20563,20565,20568],{"className":20564},[399],[63,20566,736],{"className":20567},[399,472],[63,20569,20571],{"className":20570},[476],[63,20572,20574,20594],{"className":20573},[411,412],[63,20575,20577,20591],{"className":20576},[416],[63,20578,20580],{"className":20579,"style":1532},[420],[63,20581,20582,20585],{"style":1535},[63,20583],{"className":20584,"style":817},[428],[63,20586,20588],{"className":20587},[433,434,435,436],[63,20589,14101],{"className":20590,"style":3276},[399,472,436],[63,20592,507],{"className":20593},[506],[63,20595,20597],{"className":20596},[416],[63,20598,20600],{"className":20599,"style":1554},[420],[63,20601],{},[63,20603,1078],{"className":20604},[403],[63,20606,12890],{"className":20607},[399,472],[63,20609,1083],{"className":20610},[1173],[63,20612],{"className":20613,"style":1177},[776],[63,20615,38],{"className":20616},[399,472],[63,20618,1092],{"className":20619},[593]," 表示在状态 ",[63,20622,20624,20637],{"className":20623},[342],[63,20625,20627],{"className":20626},[346],[348,20628,20629],{"xmlns":350},[352,20630,20631,20635],{},[355,20632,20633],{},[371,20634,12890],{},[378,20636,12890],{"encoding":380},[63,20638,20640],{"className":20639,"ariaHidden":386},[385],[63,20641,20643,20646],{"className":20642},[390],[63,20644],{"className":20645,"style":624},[394],[63,20647,12890],{"className":20648},[399,472]," 下采取动作 ",[63,20651,20653,20666],{"className":20652},[342],[63,20654,20656],{"className":20655},[346],[348,20657,20658],{"xmlns":350},[352,20659,20660,20664],{},[355,20661,20662],{},[371,20663,38],{},[378,20665,38],{"encoding":380},[63,20667,20669],{"className":20668,"ariaHidden":386},[385],[63,20670,20672,20675],{"className":20671},[390],[63,20673],{"className":20674,"style":624},[394],[63,20676,38],{"className":20677},[399,472]," 的期望回报，即 ",[11217,20680,20681],{},"动作价值函数","；用 ",[63,20684,20686,20710],{"className":20685},[342],[63,20687,20689],{"className":20688},[346],[348,20690,20691],{"xmlns":350},[352,20692,20693,20707],{},[355,20694,20695,20701,20703,20705],{},[368,20696,20697,20699],{},[371,20698,949],{},[371,20700,14101],{},[738,20702,1078],{"stretchy":1077},[371,20704,12890],{},[738,20706,1092],{"stretchy":1077},[378,20708,20709],{"encoding":380},"V_\\theta(s)",[63,20711,20713],{"className":20712,"ariaHidden":386},[385],[63,20714,20716,20719,20760,20763,20766],{"className":20715},[390],[63,20717],{"className":20718,"style":1155},[394],[63,20720,20722,20725],{"className":20721},[399],[63,20723,949],{"className":20724,"style":979},[399,472],[63,20726,20728],{"className":20727},[476],[63,20729,20731,20752],{"className":20730},[411,412],[63,20732,20734,20749],{"className":20733},[416],[63,20735,20737],{"className":20736,"style":1532},[420],[63,20738,20740,20743],{"style":20739},"top:-2.55em;margin-left:-0.2222em;margin-right:0.05em;",[63,20741],{"className":20742,"style":817},[428],[63,20744,20746],{"className":20745},[433,434,435,436],[63,20747,14101],{"className":20748,"style":3276},[399,472,436],[63,20750,507],{"className":20751},[506],[63,20753,20755],{"className":20754},[416],[63,20756,20758],{"className":20757,"style":1554},[420],[63,20759],{},[63,20761,1078],{"className":20762},[403],[63,20764,12890],{"className":20765},[399,472],[63,20767,1092],{"className":20768},[593],[63,20770,20772,20785],{"className":20771},[342],[63,20773,20775],{"className":20774},[346],[348,20776,20777],{"xmlns":350},[352,20778,20779,20783],{},[355,20780,20781],{},[371,20782,12890],{},[378,20784,12890],{"encoding":380},[63,20786,20788],{"className":20787,"ariaHidden":386},[385],[63,20789,20791,20794],{"className":20790},[390],[63,20792],{"className":20793,"style":624},[394],[63,20795,12890],{"className":20796},[399,472]," 下的期望回报，即 ",[11217,20799,20800],{},"状态价值函数",[14,20802,20803,20804,20807,20808,21075,21076,21104,21105,21133],{},"定义 ",[11217,20805,20806],{},"优势函数（Advantage Function）"," ",[63,20809,20811,20871],{"className":20810},[342],[63,20812,20814],{"className":20813},[346],[348,20815,20816],{"xmlns":350},[352,20817,20818,20868],{},[355,20819,20820,20826,20828,20830,20832,20834,20836,20838,20844,20846,20848,20850,20852,20854,20856,20862,20864,20866],{},[368,20821,20822,20824],{},[371,20823,1054],{},[371,20825,14101],{},[738,20827,1078],{"stretchy":1077},[371,20829,12890],{},[738,20831,1083],{"separator":386},[371,20833,38],{},[738,20835,1092],{"stretchy":1077},[738,20837,204],{},[368,20839,20840,20842],{},[371,20841,736],{},[371,20843,14101],{},[738,20845,1078],{"stretchy":1077},[371,20847,12890],{},[738,20849,1083],{"separator":386},[371,20851,38],{},[738,20853,1092],{"stretchy":1077},[738,20855,18511],{},[368,20857,20858,20860],{},[371,20859,949],{},[371,20861,14101],{},[738,20863,1078],{"stretchy":1077},[371,20865,12890],{},[738,20867,1092],{"stretchy":1077},[378,20869,20870],{"encoding":380},"A_\\theta(s,a) = Q_\\theta(s,a) - V_\\theta(s)",[63,20872,20874,20947,21020],{"className":20873,"ariaHidden":386},[385],[63,20875,20877,20880,20920,20923,20926,20929,20932,20935,20938,20941,20944],{"className":20876},[390],[63,20878],{"className":20879,"style":1155},[394],[63,20881,20883,20886],{"className":20882},[399],[63,20884,1054],{"className":20885},[399,472],[63,20887,20889],{"className":20888},[476],[63,20890,20892,20912],{"className":20891},[411,412],[63,20893,20895,20909],{"className":20894},[416],[63,20896,20898],{"className":20897,"style":1532},[420],[63,20899,20900,20903],{"style":1535},[63,20901],{"className":20902,"style":817},[428],[63,20904,20906],{"className":20905},[433,434,435,436],[63,20907,14101],{"className":20908,"style":3276},[399,472,436],[63,20910,507],{"className":20911},[506],[63,20913,20915],{"className":20914},[416],[63,20916,20918],{"className":20917,"style":1554},[420],[63,20919],{},[63,20921,1078],{"className":20922},[403],[63,20924,12890],{"className":20925},[399,472],[63,20927,1083],{"className":20928},[1173],[63,20930],{"className":20931,"style":1177},[776],[63,20933,38],{"className":20934},[399,472],[63,20936,1092],{"className":20937},[593],[63,20939],{"className":20940,"style":777},[776],[63,20942,204],{"className":20943},[781],[63,20945],{"className":20946,"style":777},[776],[63,20948,20950,20953,20993,20996,20999,21002,21005,21008,21011,21014,21017],{"className":20949},[390],[63,20951],{"className":20952,"style":1155},[394],[63,20954,20956,20959],{"className":20955},[399],[63,20957,736],{"className":20958},[399,472],[63,20960,20962],{"className":20961},[476],[63,20963,20965,20985],{"className":20964},[411,412],[63,20966,20968,20982],{"className":20967},[416],[63,20969,20971],{"className":20970,"style":1532},[420],[63,20972,20973,20976],{"style":1535},[63,20974],{"className":20975,"style":817},[428],[63,20977,20979],{"className":20978},[433,434,435,436],[63,20980,14101],{"className":20981,"style":3276},[399,472,436],[63,20983,507],{"className":20984},[506],[63,20986,20988],{"className":20987},[416],[63,20989,20991],{"className":20990,"style":1554},[420],[63,20992],{},[63,20994,1078],{"className":20995},[403],[63,20997,12890],{"className":20998},[399,472],[63,21000,1083],{"className":21001},[1173],[63,21003],{"className":21004,"style":1177},[776],[63,21006,38],{"className":21007},[399,472],[63,21009,1092],{"className":21010},[593],[63,21012],{"className":21013,"style":1920},[776],[63,21015,18511],{"className":21016},[830],[63,21018],{"className":21019,"style":1920},[776],[63,21021,21023,21026,21066,21069,21072],{"className":21022},[390],[63,21024],{"className":21025,"style":1155},[394],[63,21027,21029,21032],{"className":21028},[399],[63,21030,949],{"className":21031,"style":979},[399,472],[63,21033,21035],{"className":21034},[476],[63,21036,21038,21058],{"className":21037},[411,412],[63,21039,21041,21055],{"className":21040},[416],[63,21042,21044],{"className":21043,"style":1532},[420],[63,21045,21046,21049],{"style":20739},[63,21047],{"className":21048,"style":817},[428],[63,21050,21052],{"className":21051},[433,434,435,436],[63,21053,14101],{"className":21054,"style":3276},[399,472,436],[63,21056,507],{"className":21057},[506],[63,21059,21061],{"className":21060},[416],[63,21062,21064],{"className":21063,"style":1554},[420],[63,21065],{},[63,21067,1078],{"className":21068},[403],[63,21070,12890],{"className":21071},[399,472],[63,21073,1092],{"className":21074},[593],"，它衡量在状态 ",[63,21077,21079,21092],{"className":21078},[342],[63,21080,21082],{"className":21081},[346],[348,21083,21084],{"xmlns":350},[352,21085,21086,21090],{},[355,21087,21088],{},[371,21089,12890],{},[378,21091,12890],{"encoding":380},[63,21093,21095],{"className":21094,"ariaHidden":386},[385],[63,21096,21098,21101],{"className":21097},[390],[63,21099],{"className":21100,"style":624},[394],[63,21102,12890],{"className":21103},[399,472]," 下，动作 ",[63,21106,21108,21121],{"className":21107},[342],[63,21109,21111],{"className":21110},[346],[348,21112,21113],{"xmlns":350},[352,21114,21115,21119],{},[355,21116,21117],{},[371,21118,38],{},[378,21120,38],{"encoding":380},[63,21122,21124],{"className":21123,"ariaHidden":386},[385],[63,21125,21127,21130],{"className":21126},[390],[63,21128],{"className":21129,"style":624},[394],[63,21131,38],{"className":21132},[399,472]," 相对于\"该状态下的平均水平\"有多大优势。",[14,21135,21136],{},"将上述策略梯度公式改写为：",[14,21138,21139],{},[63,21140,21142,21256],{"className":21141},[342],[63,21143,21145],{"className":21144},[346],[348,21146,21147],{"xmlns":350},[352,21148,21149,21253],{},[355,21150,21151,21159,21173,21191,21197,21199,21207,21209,21217,21219,21221,21223,21225,21231,21233,21241,21243,21251],{},[14519,21152,21153],{"scriptlevel":1802,"displaystyle":386},[358,21154,21155,21157],{},[361,21156,363],{},[371,21158,7342],{},[1870,21160,21161,21163,21171],{},[738,21162,1874],{},[355,21164,21165,21167,21169],{},[371,21166,612],{},[738,21168,204],{},[361,21170,363],{},[371,21172,7342],{},[1870,21174,21175,21177,21185],{},[738,21176,1874],{},[355,21178,21179,21181,21183],{},[371,21180,1057],{},[738,21182,204],{},[361,21184,363],{},[368,21186,21187,21189],{},[371,21188,13358],{},[371,21190,612],{},[368,21192,21193,21195],{},[371,21194,1054],{},[371,21196,14101],{},[738,21198,1078],{"stretchy":1077},[1870,21200,21201,21203,21205],{},[371,21202,12890],{},[371,21204,1057],{},[371,21206,612],{},[738,21208,1083],{"separator":386},[1870,21210,21211,21213,21215],{},[371,21212,38],{},[371,21214,1057],{},[371,21216,612],{},[738,21218,1092],{"stretchy":1077},[371,21220,14525],{"mathvariant":1053},[371,21222,14950],{},[738,21224,1100],{},[368,21226,21227,21229],{},[371,21228,5685],{},[371,21230,14101],{},[738,21232,1078],{"stretchy":1077},[1870,21234,21235,21237,21239],{},[371,21236,38],{},[371,21238,1057],{},[371,21240,612],{},[371,21242,13030],{"mathvariant":1053},[1870,21244,21245,21247,21249],{},[371,21246,12890],{},[371,21248,1057],{},[371,21250,612],{},[738,21252,1092],{"stretchy":1077},[378,21254,21255],{"encoding":380},"\\dfrac{1}{N}\\sum\\limits_{n=1}^N \\sum\\limits_{t=1}^{T_n} A_\\theta(s_t^n,a_t^n)\\nabla \\log P_\\theta(a_t^n|s_t^n)",[63,21257,21259],{"className":21258,"ariaHidden":386},[385],[63,21260,21262,21265,21327,21330,21394,21397,21501,21504,21544,21547,21598,21601,21604,21655,21658,21661,21664,21669,21672,21712,21715,21766,21769,21820],{"className":21261},[390],[63,21263],{"className":21264,"style":18629},[394],[63,21266,21268,21271,21324],{"className":21267},[399],[63,21269],{"className":21270},[403,404],[63,21272,21274],{"className":21273},[358],[63,21275,21277,21316],{"className":21276},[411,412],[63,21278,21280,21313],{"className":21279},[416],[63,21281,21283,21294,21302],{"className":21282,"style":16424},[420],[63,21284,21285,21288],{"style":15964},[63,21286],{"className":21287,"style":429},[428],[63,21289,21291],{"className":21290},[399],[63,21292,7342],{"className":21293,"style":16436},[399,472],[63,21295,21296,21299],{"style":554},[63,21297],{"className":21298,"style":429},[428],[63,21300],{"className":21301,"style":562},[561],[63,21303,21304,21307],{"style":16030},[63,21305],{"className":21306,"style":429},[428],[63,21308,21310],{"className":21309},[399],[63,21311,363],{"className":21312},[399],[63,21314,507],{"className":21315},[506],[63,21317,21319],{"className":21318},[416],[63,21320,21322],{"className":21321,"style":16465},[420],[63,21323],{},[63,21325],{"className":21326},[593,404],[63,21328],{"className":21329,"style":1177},[776],[63,21331,21333],{"className":21332},[1212,14383],[63,21334,21336,21386],{"className":21335},[411,412],[63,21337,21339,21383],{"className":21338},[416],[63,21340,21342,21362,21372],{"className":21341,"style":18718},[420],[63,21343,21344,21347],{"style":18721},[63,21345],{"className":21346,"style":429},[428],[63,21348,21350],{"className":21349},[433,434,435,436],[63,21351,21353,21356,21359],{"className":21352},[399,436],[63,21354,612],{"className":21355},[399,472,436],[63,21357,204],{"className":21358},[781,436],[63,21360,363],{"className":21361},[399,436],[63,21363,21364,21367],{"style":458},[63,21365],{"className":21366,"style":429},[428],[63,21368,21369],{},[63,21370,1874],{"className":21371},[1212,1958,1959],[63,21373,21374,21377],{"style":18752},[63,21375],{"className":21376,"style":429},[428],[63,21378,21380],{"className":21379},[433,434,435,436],[63,21381,7342],{"className":21382,"style":16436},[399,472,436],[63,21384,507],{"className":21385},[506],[63,21387,21389],{"className":21388},[416],[63,21390,21392],{"className":21391,"style":18771},[420],[63,21393],{},[63,21395],{"className":21396,"style":1177},[776],[63,21398,21400],{"className":21399},[1212,14383],[63,21401,21403,21493],{"className":21402},[411,412],[63,21404,21406,21490],{"className":21405},[416],[63,21407,21409,21429,21439],{"className":21408,"style":18789},[420],[63,21410,21411,21414],{"style":18721},[63,21412],{"className":21413,"style":429},[428],[63,21415,21417],{"className":21416},[433,434,435,436],[63,21418,21420,21423,21426],{"className":21419},[399,436],[63,21421,1057],{"className":21422},[399,472,436],[63,21424,204],{"className":21425},[781,436],[63,21427,363],{"className":21428},[399,436],[63,21430,21431,21434],{"style":458},[63,21432],{"className":21433,"style":429},[428],[63,21435,21436],{},[63,21437,1874],{"className":21438},[1212,1958,1959],[63,21440,21441,21444],{"style":18822},[63,21442],{"className":21443,"style":429},[428],[63,21445,21447],{"className":21446},[433,434,435,436],[63,21448,21450],{"className":21449},[399,436],[63,21451,21453,21456],{"className":21452},[399,436],[63,21454,13358],{"className":21455,"style":3251},[399,472,436],[63,21457,21459],{"className":21458},[476],[63,21460,21462,21482],{"className":21461},[411,412],[63,21463,21465,21479],{"className":21464},[416],[63,21466,21468],{"className":21467,"style":4367},[420],[63,21469,21470,21473],{"style":17358},[63,21471],{"className":21472,"style":493},[428],[63,21474,21476],{"className":21475},[433,497,498,436],[63,21477,612],{"className":21478},[399,472,436],[63,21480,507],{"className":21481},[506],[63,21483,21485],{"className":21484},[416],[63,21486,21488],{"className":21487,"style":4389},[420],[63,21489],{},[63,21491,507],{"className":21492},[506],[63,21494,21496],{"className":21495},[416],[63,21497,21499],{"className":21498,"style":18771},[420],[63,21500],{},[63,21502],{"className":21503,"style":1177},[776],[63,21505,21507,21510],{"className":21506},[399],[63,21508,1054],{"className":21509},[399,472],[63,21511,21513],{"className":21512},[476],[63,21514,21516,21536],{"className":21515},[411,412],[63,21517,21519,21533],{"className":21518},[416],[63,21520,21522],{"className":21521,"style":1532},[420],[63,21523,21524,21527],{"style":1535},[63,21525],{"className":21526,"style":817},[428],[63,21528,21530],{"className":21529},[433,434,435,436],[63,21531,14101],{"className":21532,"style":3276},[399,472,436],[63,21534,507],{"className":21535},[506],[63,21537,21539],{"className":21538},[416],[63,21540,21542],{"className":21541,"style":1554},[420],[63,21543],{},[63,21545,1078],{"className":21546},[403],[63,21548,21550,21553],{"className":21549},[399],[63,21551,12890],{"className":21552},[399,472],[63,21554,21556],{"className":21555},[476],[63,21557,21559,21590],{"className":21558},[411,412],[63,21560,21562,21587],{"className":21561},[416],[63,21563,21565,21576],{"className":21564,"style":16768},[420],[63,21566,21567,21570],{"style":17457},[63,21568],{"className":21569,"style":817},[428],[63,21571,21573],{"className":21572},[433,434,435,436],[63,21574,1057],{"className":21575},[399,472,436],[63,21577,21578,21581],{"style":813},[63,21579],{"className":21580,"style":817},[428],[63,21582,21584],{"className":21583},[433,434,435,436],[63,21585,612],{"className":21586},[399,472,436],[63,21588,507],{"className":21589},[506],[63,21591,21593],{"className":21592},[416],[63,21594,21596],{"className":21595,"style":17487},[420],[63,21597],{},[63,21599,1083],{"className":21600},[1173],[63,21602],{"className":21603,"style":1177},[776],[63,21605,21607,21610],{"className":21606},[399],[63,21608,38],{"className":21609},[399,472],[63,21611,21613],{"className":21612},[476],[63,21614,21616,21647],{"className":21615},[411,412],[63,21617,21619,21644],{"className":21618},[416],[63,21620,21622,21633],{"className":21621,"style":16768},[420],[63,21623,21624,21627],{"style":17457},[63,21625],{"className":21626,"style":817},[428],[63,21628,21630],{"className":21629},[433,434,435,436],[63,21631,1057],{"className":21632},[399,472,436],[63,21634,21635,21638],{"style":813},[63,21636],{"className":21637,"style":817},[428],[63,21639,21641],{"className":21640},[433,434,435,436],[63,21642,612],{"className":21643},[399,472,436],[63,21645,507],{"className":21646},[506],[63,21648,21650],{"className":21649},[416],[63,21651,21653],{"className":21652,"style":17487},[420],[63,21654],{},[63,21656,1092],{"className":21657},[593],[63,21659,14525],{"className":21660},[399],[63,21662],{"className":21663,"style":1177},[776],[63,21665,16996,21667],{"className":21666},[1212],[63,21668,17000],{"style":16999},[63,21670],{"className":21671,"style":1177},[776],[63,21673,21675,21678],{"className":21674},[399],[63,21676,5685],{"className":21677,"style":3251},[399,472],[63,21679,21681],{"className":21680},[476],[63,21682,21684,21704],{"className":21683},[411,412],[63,21685,21687,21701],{"className":21686},[416],[63,21688,21690],{"className":21689,"style":1532},[420],[63,21691,21692,21695],{"style":7483},[63,21693],{"className":21694,"style":817},[428],[63,21696,21698],{"className":21697},[433,434,435,436],[63,21699,14101],{"className":21700,"style":3276},[399,472,436],[63,21702,507],{"className":21703},[506],[63,21705,21707],{"className":21706},[416],[63,21708,21710],{"className":21709,"style":1554},[420],[63,21711],{},[63,21713,1078],{"className":21714},[403],[63,21716,21718,21721],{"className":21717},[399],[63,21719,38],{"className":21720},[399,472],[63,21722,21724],{"className":21723},[476],[63,21725,21727,21758],{"className":21726},[411,412],[63,21728,21730,21755],{"className":21729},[416],[63,21731,21733,21744],{"className":21732,"style":16768},[420],[63,21734,21735,21738],{"style":17457},[63,21736],{"className":21737,"style":817},[428],[63,21739,21741],{"className":21740},[433,434,435,436],[63,21742,1057],{"className":21743},[399,472,436],[63,21745,21746,21749],{"style":813},[63,21747],{"className":21748,"style":817},[428],[63,21750,21752],{"className":21751},[433,434,435,436],[63,21753,612],{"className":21754},[399,472,436],[63,21756,507],{"className":21757},[506],[63,21759,21761],{"className":21760},[416],[63,21762,21764],{"className":21763,"style":17487},[420],[63,21765],{},[63,21767,13030],{"className":21768},[399],[63,21770,21772,21775],{"className":21771},[399],[63,21773,12890],{"className":21774},[399,472],[63,21776,21778],{"className":21777},[476],[63,21779,21781,21812],{"className":21780},[411,412],[63,21782,21784,21809],{"className":21783},[416],[63,21785,21787,21798],{"className":21786,"style":16768},[420],[63,21788,21789,21792],{"style":17457},[63,21790],{"className":21791,"style":817},[428],[63,21793,21795],{"className":21794},[433,434,435,436],[63,21796,1057],{"className":21797},[399,472,436],[63,21799,21800,21803],{"style":813},[63,21801],{"className":21802,"style":817},[428],[63,21804,21806],{"className":21805},[433,434,435,436],[63,21807,612],{"className":21808},[399,472,436],[63,21810,507],{"className":21811},[506],[63,21813,21815],{"className":21814},[416],[63,21816,21818],{"className":21817,"style":17487},[420],[63,21819],{},[63,21821,1092],{"className":21822},[593],[14,21824,21825],{},"利用 Bellman 关系，可以把动作价值函数与状态价值函数联系起来：",[63,21827,21829],{"className":21828},[14495],[63,21830,21832,22037],{"className":21831},[342],[63,21833,21835],{"className":21834},[346],[348,21836,21837],{"xmlns":350,"display":9561},[352,21838,21839,22034],{},[355,21840,21841,21847,21849,21855,21857,21859,21861,21863,21869,21871,21873,21879,21881,21893,21895,21898,21904,21906,21912,21914,21916,21918,21920,21926,21928,21930,21936,21938,21950,21952,21954,21960,21962,21968,21970,21972,21978,21980,21992,21994,21996,22008,22010,22012,22018,22020,22032],{},[368,21842,21843,21845],{},[371,21844,736],{},[371,21846,14101],{},[738,21848,1078],{"stretchy":1077},[368,21850,21851,21853],{},[371,21852,12890],{},[371,21854,1057],{},[738,21856,1083],{"separator":386},[371,21858,38],{},[738,21860,1092],{"stretchy":1077},[738,21862,204],{},[368,21864,21865,21867],{},[371,21866,7852],{},[371,21868,1057],{},[738,21870,5839],{},[371,21872,19142],{},[368,21874,21875,21877],{},[371,21876,949],{},[371,21878,14101],{},[738,21880,1078],{"stretchy":1077},[368,21882,21883,21885],{},[371,21884,12890],{},[355,21886,21887,21889,21891],{},[371,21888,1057],{},[738,21890,5839],{},[361,21892,363],{},[738,21894,1092],{"stretchy":1077},[776,21896],{"linebreak":21897},"newline",[368,21899,21900,21902],{},[371,21901,1054],{},[371,21903,14101],{},[738,21905,1078],{"stretchy":1077},[368,21907,21908,21910],{},[371,21909,12890],{},[371,21911,1057],{},[738,21913,1083],{"separator":386},[371,21915,38],{},[738,21917,1092],{"stretchy":1077},[738,21919,204],{},[368,21921,21922,21924],{},[371,21923,7852],{},[371,21925,1057],{},[738,21927,5839],{},[371,21929,19142],{},[368,21931,21932,21934],{},[371,21933,949],{},[371,21935,14101],{},[738,21937,1078],{"stretchy":1077},[368,21939,21940,21942],{},[371,21941,12890],{},[355,21943,21944,21946,21948],{},[371,21945,1057],{},[738,21947,5839],{},[361,21949,363],{},[738,21951,1092],{"stretchy":1077},[738,21953,18511],{},[368,21955,21956,21958],{},[371,21957,949],{},[371,21959,14101],{},[738,21961,1078],{"stretchy":1077},[368,21963,21964,21966],{},[371,21965,12890],{},[371,21967,1057],{},[738,21969,1092],{"stretchy":1077},[776,21971],{"linebreak":21897},[368,21973,21974,21976],{},[371,21975,949],{},[371,21977,14101],{},[738,21979,1078],{"stretchy":1077},[368,21981,21982,21984],{},[371,21983,12890],{},[355,21985,21986,21988,21990],{},[371,21987,1057],{},[738,21989,5839],{},[361,21991,363],{},[738,21993,1092],{"stretchy":1077},[738,21995,14820],{},[368,21997,21998,22000],{},[371,21999,7852],{},[355,22001,22002,22004,22006],{},[371,22003,1057],{},[738,22005,5839],{},[361,22007,363],{},[738,22009,5839],{},[371,22011,19142],{},[368,22013,22014,22016],{},[371,22015,949],{},[371,22017,14101],{},[738,22019,1078],{"stretchy":1077},[368,22021,22022,22024],{},[371,22023,12890],{},[355,22025,22026,22028,22030],{},[371,22027,1057],{},[738,22029,5839],{},[361,22031,2883],{},[738,22033,1092],{"stretchy":1077},[378,22035,22036],{"encoding":380}," Q_\\theta(s_t, a) = r_t + \\gamma V_\\theta(s_{t+1}) \\\\\n\n A_\\theta(s_t, a) = r_t + \\gamma V_\\theta(s_{t+1}) - V_\\theta(s_t)\\\\\n V_\\theta(s_{t+1}) \\approx r_{t+1} + \\gamma V_\\theta(s_{t+2})",[63,22038,22040,22150,22206,22310,22313,22423,22478,22591,22683,22686,22796,22861],{"className":22039,"ariaHidden":386},[385],[63,22041,22043,22046,22086,22089,22129,22132,22135,22138,22141,22144,22147],{"className":22042},[390],[63,22044],{"className":22045,"style":1155},[394],[63,22047,22049,22052],{"className":22048},[399],[63,22050,736],{"className":22051},[399,472],[63,22053,22055],{"className":22054},[476],[63,22056,22058,22078],{"className":22057},[411,412],[63,22059,22061,22075],{"className":22060},[416],[63,22062,22064],{"className":22063,"style":1532},[420],[63,22065,22066,22069],{"style":1535},[63,22067],{"className":22068,"style":817},[428],[63,22070,22072],{"className":22071},[433,434,435,436],[63,22073,14101],{"className":22074,"style":3276},[399,472,436],[63,22076,507],{"className":22077},[506],[63,22079,22081],{"className":22080},[416],[63,22082,22084],{"className":22083,"style":1554},[420],[63,22085],{},[63,22087,1078],{"className":22088},[403],[63,22090,22092,22095],{"className":22091},[399],[63,22093,12890],{"className":22094},[399,472],[63,22096,22098],{"className":22097},[476],[63,22099,22101,22121],{"className":22100},[411,412],[63,22102,22104,22118],{"className":22103},[416],[63,22105,22107],{"className":22106,"style":12818},[420],[63,22108,22109,22112],{"style":1535},[63,22110],{"className":22111,"style":817},[428],[63,22113,22115],{"className":22114},[433,434,435,436],[63,22116,1057],{"className":22117},[399,472,436],[63,22119,507],{"className":22120},[506],[63,22122,22124],{"className":22123},[416],[63,22125,22127],{"className":22126,"style":1554},[420],[63,22128],{},[63,22130,1083],{"className":22131},[1173],[63,22133],{"className":22134,"style":1177},[776],[63,22136,38],{"className":22137},[399,472],[63,22139,1092],{"className":22140},[593],[63,22142],{"className":22143,"style":777},[776],[63,22145,204],{"className":22146},[781],[63,22148],{"className":22149,"style":777},[776],[63,22151,22153,22157,22197,22200,22203],{"className":22152},[390],[63,22154],{"className":22155,"style":22156},[394],"height:0.7333em;vertical-align:-0.15em;",[63,22158,22160,22163],{"className":22159},[399],[63,22161,7852],{"className":22162,"style":3276},[399,472],[63,22164,22166],{"className":22165},[476],[63,22167,22169,22189],{"className":22168},[411,412],[63,22170,22172,22186],{"className":22171},[416],[63,22173,22175],{"className":22174,"style":12818},[420],[63,22176,22177,22180],{"style":13208},[63,22178],{"className":22179,"style":817},[428],[63,22181,22183],{"className":22182},[433,434,435,436],[63,22184,1057],{"className":22185},[399,472,436],[63,22187,507],{"className":22188},[506],[63,22190,22192],{"className":22191},[416],[63,22193,22195],{"className":22194,"style":1554},[420],[63,22196],{},[63,22198],{"className":22199,"style":1920},[776],[63,22201,5839],{"className":22202},[830],[63,22204],{"className":22205,"style":1920},[776],[63,22207,22209,22212,22215,22255,22258,22307],{"className":22208},[390],[63,22210],{"className":22211,"style":1155},[394],[63,22213,19142],{"className":22214,"style":19389},[399,472],[63,22216,22218,22221],{"className":22217},[399],[63,22219,949],{"className":22220,"style":979},[399,472],[63,22222,22224],{"className":22223},[476],[63,22225,22227,22247],{"className":22226},[411,412],[63,22228,22230,22244],{"className":22229},[416],[63,22231,22233],{"className":22232,"style":1532},[420],[63,22234,22235,22238],{"style":20739},[63,22236],{"className":22237,"style":817},[428],[63,22239,22241],{"className":22240},[433,434,435,436],[63,22242,14101],{"className":22243,"style":3276},[399,472,436],[63,22245,507],{"className":22246},[506],[63,22248,22250],{"className":22249},[416],[63,22251,22253],{"className":22252,"style":1554},[420],[63,22254],{},[63,22256,1078],{"className":22257},[403],[63,22259,22261,22264],{"className":22260},[399],[63,22262,12890],{"className":22263},[399,472],[63,22265,22267],{"className":22266},[476],[63,22268,22270,22299],{"className":22269},[411,412],[63,22271,22273,22296],{"className":22272},[416],[63,22274,22276],{"className":22275,"style":3157},[420],[63,22277,22278,22281],{"style":1535},[63,22279],{"className":22280,"style":817},[428],[63,22282,22284],{"className":22283},[433,434,435,436],[63,22285,22287,22290,22293],{"className":22286},[399,436],[63,22288,1057],{"className":22289},[399,472,436],[63,22291,5839],{"className":22292},[830,436],[63,22294,363],{"className":22295},[399,436],[63,22297,507],{"className":22298},[506],[63,22300,22302],{"className":22301},[416],[63,22303,22305],{"className":22304,"style":6959},[420],[63,22306],{},[63,22308,1092],{"className":22309},[593],[63,22311],{"className":22312},[776,21897],[63,22314,22316,22319,22359,22362,22402,22405,22408,22411,22414,22417,22420],{"className":22315},[390],[63,22317],{"className":22318,"style":1155},[394],[63,22320,22322,22325],{"className":22321},[399],[63,22323,1054],{"className":22324},[399,472],[63,22326,22328],{"className":22327},[476],[63,22329,22331,22351],{"className":22330},[411,412],[63,22332,22334,22348],{"className":22333},[416],[63,22335,22337],{"className":22336,"style":1532},[420],[63,22338,22339,22342],{"style":1535},[63,22340],{"className":22341,"style":817},[428],[63,22343,22345],{"className":22344},[433,434,435,436],[63,22346,14101],{"className":22347,"style":3276},[399,472,436],[63,22349,507],{"className":22350},[506],[63,22352,22354],{"className":22353},[416],[63,22355,22357],{"className":22356,"style":1554},[420],[63,22358],{},[63,22360,1078],{"className":22361},[403],[63,22363,22365,22368],{"className":22364},[399],[63,22366,12890],{"className":22367},[399,472],[63,22369,22371],{"className":22370},[476],[63,22372,22374,22394],{"className":22373},[411,412],[63,22375,22377,22391],{"className":22376},[416],[63,22378,22380],{"className":22379,"style":12818},[420],[63,22381,22382,22385],{"style":1535},[63,22383],{"className":22384,"style":817},[428],[63,22386,22388],{"className":22387},[433,434,435,436],[63,22389,1057],{"className":22390},[399,472,436],[63,22392,507],{"className":22393},[506],[63,22395,22397],{"className":22396},[416],[63,22398,22400],{"className":22399,"style":1554},[420],[63,22401],{},[63,22403,1083],{"className":22404},[1173],[63,22406],{"className":22407,"style":1177},[776],[63,22409,38],{"className":22410},[399,472],[63,22412,1092],{"className":22413},[593],[63,22415],{"className":22416,"style":777},[776],[63,22418,204],{"className":22419},[781],[63,22421],{"className":22422,"style":777},[776],[63,22424,22426,22429,22469,22472,22475],{"className":22425},[390],[63,22427],{"className":22428,"style":22156},[394],[63,22430,22432,22435],{"className":22431},[399],[63,22433,7852],{"className":22434,"style":3276},[399,472],[63,22436,22438],{"className":22437},[476],[63,22439,22441,22461],{"className":22440},[411,412],[63,22442,22444,22458],{"className":22443},[416],[63,22445,22447],{"className":22446,"style":12818},[420],[63,22448,22449,22452],{"style":13208},[63,22450],{"className":22451,"style":817},[428],[63,22453,22455],{"className":22454},[433,434,435,436],[63,22456,1057],{"className":22457},[399,472,436],[63,22459,507],{"className":22460},[506],[63,22462,22464],{"className":22463},[416],[63,22465,22467],{"className":22466,"style":1554},[420],[63,22468],{},[63,22470],{"className":22471,"style":1920},[776],[63,22473,5839],{"className":22474},[830],[63,22476],{"className":22477,"style":1920},[776],[63,22479,22481,22484,22487,22527,22530,22579,22582,22585,22588],{"className":22480},[390],[63,22482],{"className":22483,"style":1155},[394],[63,22485,19142],{"className":22486,"style":19389},[399,472],[63,22488,22490,22493],{"className":22489},[399],[63,22491,949],{"className":22492,"style":979},[399,472],[63,22494,22496],{"className":22495},[476],[63,22497,22499,22519],{"className":22498},[411,412],[63,22500,22502,22516],{"className":22501},[416],[63,22503,22505],{"className":22504,"style":1532},[420],[63,22506,22507,22510],{"style":20739},[63,22508],{"className":22509,"style":817},[428],[63,22511,22513],{"className":22512},[433,434,435,436],[63,22514,14101],{"className":22515,"style":3276},[399,472,436],[63,22517,507],{"className":22518},[506],[63,22520,22522],{"className":22521},[416],[63,22523,22525],{"className":22524,"style":1554},[420],[63,22526],{},[63,22528,1078],{"className":22529},[403],[63,22531,22533,22536],{"className":22532},[399],[63,22534,12890],{"className":22535},[399,472],[63,22537,22539],{"className":22538},[476],[63,22540,22542,22571],{"className":22541},[411,412],[63,22543,22545,22568],{"className":22544},[416],[63,22546,22548],{"className":22547,"style":3157},[420],[63,22549,22550,22553],{"style":1535},[63,22551],{"className":22552,"style":817},[428],[63,22554,22556],{"className":22555},[433,434,435,436],[63,22557,22559,22562,22565],{"className":22558},[399,436],[63,22560,1057],{"className":22561},[399,472,436],[63,22563,5839],{"className":22564},[830,436],[63,22566,363],{"className":22567},[399,436],[63,22569,507],{"className":22570},[506],[63,22572,22574],{"className":22573},[416],[63,22575,22577],{"className":22576,"style":6959},[420],[63,22578],{},[63,22580,1092],{"className":22581},[593],[63,22583],{"className":22584,"style":1920},[776],[63,22586,18511],{"className":22587},[830],[63,22589],{"className":22590,"style":1920},[776],[63,22592,22594,22597,22637,22640,22680],{"className":22593},[390],[63,22595],{"className":22596,"style":1155},[394],[63,22598,22600,22603],{"className":22599},[399],[63,22601,949],{"className":22602,"style":979},[399,472],[63,22604,22606],{"className":22605},[476],[63,22607,22609,22629],{"className":22608},[411,412],[63,22610,22612,22626],{"className":22611},[416],[63,22613,22615],{"className":22614,"style":1532},[420],[63,22616,22617,22620],{"style":20739},[63,22618],{"className":22619,"style":817},[428],[63,22621,22623],{"className":22622},[433,434,435,436],[63,22624,14101],{"className":22625,"style":3276},[399,472,436],[63,22627,507],{"className":22628},[506],[63,22630,22632],{"className":22631},[416],[63,22633,22635],{"className":22634,"style":1554},[420],[63,22636],{},[63,22638,1078],{"className":22639},[403],[63,22641,22643,22646],{"className":22642},[399],[63,22644,12890],{"className":22645},[399,472],[63,22647,22649],{"className":22648},[476],[63,22650,22652,22672],{"className":22651},[411,412],[63,22653,22655,22669],{"className":22654},[416],[63,22656,22658],{"className":22657,"style":12818},[420],[63,22659,22660,22663],{"style":1535},[63,22661],{"className":22662,"style":817},[428],[63,22664,22666],{"className":22665},[433,434,435,436],[63,22667,1057],{"className":22668},[399,472,436],[63,22670,507],{"className":22671},[506],[63,22673,22675],{"className":22674},[416],[63,22676,22678],{"className":22677,"style":1554},[420],[63,22679],{},[63,22681,1092],{"className":22682},[593],[63,22684],{"className":22685},[776,21897],[63,22687,22689,22692,22732,22735,22784,22787,22790,22793],{"className":22688},[390],[63,22690],{"className":22691,"style":1155},[394],[63,22693,22695,22698],{"className":22694},[399],[63,22696,949],{"className":22697,"style":979},[399,472],[63,22699,22701],{"className":22700},[476],[63,22702,22704,22724],{"className":22703},[411,412],[63,22705,22707,22721],{"className":22706},[416],[63,22708,22710],{"className":22709,"style":1532},[420],[63,22711,22712,22715],{"style":20739},[63,22713],{"className":22714,"style":817},[428],[63,22716,22718],{"className":22717},[433,434,435,436],[63,22719,14101],{"className":22720,"style":3276},[399,472,436],[63,22722,507],{"className":22723},[506],[63,22725,22727],{"className":22726},[416],[63,22728,22730],{"className":22729,"style":1554},[420],[63,22731],{},[63,22733,1078],{"className":22734},[403],[63,22736,22738,22741],{"className":22737},[399],[63,22739,12890],{"className":22740},[399,472],[63,22742,22744],{"className":22743},[476],[63,22745,22747,22776],{"className":22746},[411,412],[63,22748,22750,22773],{"className":22749},[416],[63,22751,22753],{"className":22752,"style":3157},[420],[63,22754,22755,22758],{"style":1535},[63,22756],{"className":22757,"style":817},[428],[63,22759,22761],{"className":22760},[433,434,435,436],[63,22762,22764,22767,22770],{"className":22763},[399,436],[63,22765,1057],{"className":22766},[399,472,436],[63,22768,5839],{"className":22769},[830,436],[63,22771,363],{"className":22772},[399,436],[63,22774,507],{"className":22775},[506],[63,22777,22779],{"className":22778},[416],[63,22780,22782],{"className":22781,"style":6959},[420],[63,22783],{},[63,22785,1092],{"className":22786},[593],[63,22788],{"className":22789,"style":777},[776],[63,22791,14820],{"className":22792},[781],[63,22794],{"className":22795,"style":777},[776],[63,22797,22799,22803,22852,22855,22858],{"className":22798},[390],[63,22800],{"className":22801,"style":22802},[394],"height:0.7917em;vertical-align:-0.2083em;",[63,22804,22806,22809],{"className":22805},[399],[63,22807,7852],{"className":22808,"style":3276},[399,472],[63,22810,22812],{"className":22811},[476],[63,22813,22815,22844],{"className":22814},[411,412],[63,22816,22818,22841],{"className":22817},[416],[63,22819,22821],{"className":22820,"style":3157},[420],[63,22822,22823,22826],{"style":13208},[63,22824],{"className":22825,"style":817},[428],[63,22827,22829],{"className":22828},[433,434,435,436],[63,22830,22832,22835,22838],{"className":22831},[399,436],[63,22833,1057],{"className":22834},[399,472,436],[63,22836,5839],{"className":22837},[830,436],[63,22839,363],{"className":22840},[399,436],[63,22842,507],{"className":22843},[506],[63,22845,22847],{"className":22846},[416],[63,22848,22850],{"className":22849,"style":6959},[420],[63,22851],{},[63,22853],{"className":22854,"style":1920},[776],[63,22856,5839],{"className":22857},[830],[63,22859],{"className":22860,"style":1920},[776],[63,22862,22864,22867,22870,22910,22913,22962],{"className":22863},[390],[63,22865],{"className":22866,"style":1155},[394],[63,22868,19142],{"className":22869,"style":19389},[399,472],[63,22871,22873,22876],{"className":22872},[399],[63,22874,949],{"className":22875,"style":979},[399,472],[63,22877,22879],{"className":22878},[476],[63,22880,22882,22902],{"className":22881},[411,412],[63,22883,22885,22899],{"className":22884},[416],[63,22886,22888],{"className":22887,"style":1532},[420],[63,22889,22890,22893],{"style":20739},[63,22891],{"className":22892,"style":817},[428],[63,22894,22896],{"className":22895},[433,434,435,436],[63,22897,14101],{"className":22898,"style":3276},[399,472,436],[63,22900,507],{"className":22901},[506],[63,22903,22905],{"className":22904},[416],[63,22906,22908],{"className":22907,"style":1554},[420],[63,22909],{},[63,22911,1078],{"className":22912},[403],[63,22914,22916,22919],{"className":22915},[399],[63,22917,12890],{"className":22918},[399,472],[63,22920,22922],{"className":22921},[476],[63,22923,22925,22954],{"className":22924},[411,412],[63,22926,22928,22951],{"className":22927},[416],[63,22929,22931],{"className":22930,"style":3157},[420],[63,22932,22933,22936],{"style":1535},[63,22934],{"className":22935,"style":817},[428],[63,22937,22939],{"className":22938},[433,434,435,436],[63,22940,22942,22945,22948],{"className":22941},[399,436],[63,22943,1057],{"className":22944},[399,472,436],[63,22946,5839],{"className":22947},[830,436],[63,22949,2883],{"className":22950},[399,436],[63,22952,507],{"className":22953},[506],[63,22955,22957],{"className":22956},[416],[63,22958,22960],{"className":22959,"style":6959},[420],[63,22961],{},[63,22963,1092],{"className":22964},[593],[14,22966,22967,22968,23066,23067,23137],{},"经过这一步代换，",[63,22969,22971,22999],{"className":22970},[342],[63,22972,22974],{"className":22973},[346],[348,22975,22976],{"xmlns":350},[352,22977,22978,22996],{},[355,22979,22980,22986,22988,22990,22992,22994],{},[368,22981,22982,22984],{},[371,22983,1054],{},[371,22985,14101],{},[738,22987,1078],{"stretchy":1077},[371,22989,12890],{},[738,22991,1083],{"separator":386},[371,22993,38],{},[738,22995,1092],{"stretchy":1077},[378,22997,22998],{"encoding":380},"A_\\theta(s,a)",[63,23000,23002],{"className":23001,"ariaHidden":386},[385],[63,23003,23005,23008,23048,23051,23054,23057,23060,23063],{"className":23004},[390],[63,23006],{"className":23007,"style":1155},[394],[63,23009,23011,23014],{"className":23010},[399],[63,23012,1054],{"className":23013},[399,472],[63,23015,23017],{"className":23016},[476],[63,23018,23020,23040],{"className":23019},[411,412],[63,23021,23023,23037],{"className":23022},[416],[63,23024,23026],{"className":23025,"style":1532},[420],[63,23027,23028,23031],{"style":1535},[63,23029],{"className":23030,"style":817},[428],[63,23032,23034],{"className":23033},[433,434,435,436],[63,23035,14101],{"className":23036,"style":3276},[399,472,436],[63,23038,507],{"className":23039},[506],[63,23041,23043],{"className":23042},[416],[63,23044,23046],{"className":23045,"style":1554},[420],[63,23047],{},[63,23049,1078],{"className":23050},[403],[63,23052,12890],{"className":23053},[399,472],[63,23055,1083],{"className":23056},[1173],[63,23058],{"className":23059,"style":1177},[776],[63,23061,38],{"className":23062},[399,472],[63,23064,1092],{"className":23065},[593]," 的估计只依赖一个价值函数 ",[63,23068,23070,23088],{"className":23069},[342],[63,23071,23073],{"className":23072},[346],[348,23074,23075],{"xmlns":350},[352,23076,23077,23085],{},[355,23078,23079],{},[368,23080,23081,23083],{},[371,23082,949],{},[371,23084,14101],{},[378,23086,23087],{"encoding":380},"V_\\theta",[63,23089,23091],{"className":23090,"ariaHidden":386},[385],[63,23092,23094,23097],{"className":23093},[390],[63,23095],{"className":23096,"style":6995},[394],[63,23098,23100,23103],{"className":23099},[399],[63,23101,949],{"className":23102,"style":979},[399,472],[63,23104,23106],{"className":23105},[476],[63,23107,23109,23129],{"className":23108},[411,412],[63,23110,23112,23126],{"className":23111},[416],[63,23113,23115],{"className":23114,"style":1532},[420],[63,23116,23117,23120],{"style":20739},[63,23118],{"className":23119,"style":817},[428],[63,23121,23123],{"className":23122},[433,434,435,436],[63,23124,14101],{"className":23125,"style":3276},[399,472,436],[63,23127,507],{"className":23128},[506],[63,23130,23132],{"className":23131},[416],[63,23133,23135],{"className":23134,"style":1554},[420],[63,23136],{},"，公式整体的复杂度也得以降低。",[14,23139,23140,23141,23210],{},"那么应该向后展开（rollout）多少步再用 ",[63,23142,23144,23161],{"className":23143},[342],[63,23145,23147],{"className":23146},[346],[348,23148,23149],{"xmlns":350},[352,23150,23151,23159],{},[355,23152,23153],{},[368,23154,23155,23157],{},[371,23156,949],{},[371,23158,14101],{},[378,23160,23087],{"encoding":380},[63,23162,23164],{"className":23163,"ariaHidden":386},[385],[63,23165,23167,23170],{"className":23166},[390],[63,23168],{"className":23169,"style":6995},[394],[63,23171,23173,23176],{"className":23172},[399],[63,23174,949],{"className":23175,"style":979},[399,472],[63,23177,23179],{"className":23178},[476],[63,23180,23182,23202],{"className":23181},[411,412],[63,23183,23185,23199],{"className":23184},[416],[63,23186,23188],{"className":23187,"style":1532},[420],[63,23189,23190,23193],{"style":20739},[63,23191],{"className":23192,"style":817},[428],[63,23194,23196],{"className":23195},[433,434,435,436],[63,23197,14101],{"className":23198,"style":3276},[399,472,436],[63,23200,507],{"className":23201},[506],[63,23203,23205],{"className":23204},[416],[63,23206,23208],{"className":23207,"style":1554},[420],[63,23209],{}," 截断呢？展开越多步，估计就越接近真实回报，偏差越小，但方差也会越大；反之展开越少，偏差大、方差小。这就是经典的 bias-variance trade-off：",[63,23212,23214],{"className":23213},[14495],[63,23215,23217,23667],{"className":23216},[342],[63,23218,23220],{"className":23219},[346],[348,23221,23222],{"xmlns":350,"display":9561},[352,23223,23224,23664],{},[355,23225,23226,23234,23236,23242,23244,23246,23248,23250,23256,23258,23260,23266,23268,23280,23282,23284,23290,23292,23298,23300,23302,23310,23312,23318,23320,23322,23324,23326,23332,23334,23336,23348,23350,23356,23362,23364,23376,23378,23380,23386,23388,23394,23396,23398,23406,23408,23414,23416,23418,23420,23422,23428,23430,23432,23435,23447,23449,23455,23457,23469,23471,23477,23483,23485,23497,23499,23501,23507,23509,23515,23517,23519,23531,23533,23541,23543,23549,23551,23553,23555,23557,23563,23565,23567,23569,23581,23583,23589,23591,23603,23605,23611,23613,23625,23627,23630,23632,23638,23640,23646,23648,23654,23656,23662],{},[1870,23227,23228,23230,23232],{},[371,23229,1054],{},[371,23231,14101],{},[361,23233,363],{},[738,23235,1078],{"stretchy":1077},[368,23237,23238,23240],{},[371,23239,12890],{},[371,23241,1057],{},[738,23243,1083],{"separator":386},[371,23245,38],{},[738,23247,1092],{"stretchy":1077},[738,23249,204],{},[368,23251,23252,23254],{},[371,23253,7852],{},[371,23255,1057],{},[738,23257,5839],{},[371,23259,19142],{},[368,23261,23262,23264],{},[371,23263,949],{},[371,23265,14101],{},[738,23267,1078],{"stretchy":1077},[368,23269,23270,23272],{},[371,23271,12890],{},[355,23273,23274,23276,23278],{},[371,23275,1057],{},[738,23277,5839],{},[361,23279,363],{},[738,23281,1092],{"stretchy":1077},[738,23283,18511],{},[368,23285,23286,23288],{},[371,23287,949],{},[371,23289,14101],{},[738,23291,1078],{"stretchy":1077},[368,23293,23294,23296],{},[371,23295,12890],{},[371,23297,1057],{},[738,23299,1092],{"stretchy":1077},[776,23301],{"linebreak":21897},[1870,23303,23304,23306,23308],{},[371,23305,1054],{},[371,23307,14101],{},[361,23309,2883],{},[738,23311,1078],{"stretchy":1077},[368,23313,23314,23316],{},[371,23315,12890],{},[371,23317,1057],{},[738,23319,1083],{"separator":386},[371,23321,38],{},[738,23323,1092],{"stretchy":1077},[738,23325,204],{},[368,23327,23328,23330],{},[371,23329,7852],{},[371,23331,1057],{},[738,23333,5839],{},[371,23335,19142],{},[368,23337,23338,23340],{},[371,23339,7852],{},[355,23341,23342,23344,23346],{},[371,23343,1057],{},[738,23345,5839],{},[361,23347,363],{},[738,23349,5839],{},[742,23351,23352,23354],{},[371,23353,19142],{},[361,23355,2883],{},[368,23357,23358,23360],{},[371,23359,949],{},[371,23361,14101],{},[738,23363,1078],{"stretchy":1077},[368,23365,23366,23368],{},[371,23367,12890],{},[355,23369,23370,23372,23374],{},[371,23371,1057],{},[738,23373,5839],{},[361,23375,2883],{},[738,23377,1092],{"stretchy":1077},[738,23379,18511],{},[368,23381,23382,23384],{},[371,23383,949],{},[371,23385,14101],{},[738,23387,1078],{"stretchy":1077},[368,23389,23390,23392],{},[371,23391,12890],{},[371,23393,1057],{},[738,23395,1092],{"stretchy":1077},[776,23397],{"linebreak":21897},[1870,23399,23400,23402,23404],{},[371,23401,1054],{},[371,23403,14101],{},[361,23405,4738],{},[738,23407,1078],{"stretchy":1077},[368,23409,23410,23412],{},[371,23411,12890],{},[371,23413,1057],{},[738,23415,1083],{"separator":386},[371,23417,38],{},[738,23419,1092],{"stretchy":1077},[738,23421,204],{},[368,23423,23424,23426],{},[371,23425,7852],{},[371,23427,1057],{},[738,23429,5839],{},[371,23431,19142],{},[738,23433,23434],{},"∗",[368,23436,23437,23439],{},[371,23438,7852],{},[355,23440,23441,23443,23445],{},[371,23442,1057],{},[738,23444,5839],{},[361,23446,363],{},[738,23448,5839],{},[742,23450,23451,23453],{},[371,23452,19142],{},[361,23454,2883],{},[738,23456,23434],{},[368,23458,23459,23461],{},[371,23460,7852],{},[355,23462,23463,23465,23467],{},[371,23464,1057],{},[738,23466,5839],{},[361,23468,2883],{},[738,23470,5839],{},[742,23472,23473,23475],{},[371,23474,19142],{},[361,23476,4738],{},[368,23478,23479,23481],{},[371,23480,949],{},[371,23482,14101],{},[738,23484,1078],{"stretchy":1077},[368,23486,23487,23489],{},[371,23488,12890],{},[355,23490,23491,23493,23495],{},[371,23492,1057],{},[738,23494,5839],{},[361,23496,4738],{},[738,23498,1092],{"stretchy":1077},[738,23500,18511],{},[368,23502,23503,23505],{},[371,23504,949],{},[371,23506,14101],{},[738,23508,1078],{"stretchy":1077},[368,23510,23511,23513],{},[371,23512,12890],{},[371,23514,1057],{},[738,23516,1092],{"stretchy":1077},[776,23518],{"linebreak":21897},[355,23520,23521,23524],{},[371,23522,23523],{"mathvariant":1053},"⋮",[23525,23526,23527],"mpadded",{"height":14511,"voffset":14511},[776,23528],{"mathbackground":23529,"width":14511,"height":23530},"black","1.5em",[776,23532],{"linebreak":21897},[1870,23534,23535,23537,23539],{},[371,23536,1054],{},[371,23538,14101],{},[371,23540,13358],{},[738,23542,1078],{"stretchy":1077},[368,23544,23545,23547],{},[371,23546,12890],{},[371,23548,1057],{},[738,23550,1083],{"separator":386},[371,23552,38],{},[738,23554,1092],{"stretchy":1077},[738,23556,204],{},[368,23558,23559,23561],{},[371,23560,7852],{},[371,23562,1057],{},[738,23564,5839],{},[371,23566,19142],{},[738,23568,23434],{},[368,23570,23571,23573],{},[371,23572,7852],{},[355,23574,23575,23577,23579],{},[371,23576,1057],{},[738,23578,5839],{},[361,23580,363],{},[738,23582,5839],{},[742,23584,23585,23587],{},[371,23586,19142],{},[361,23588,2883],{},[738,23590,23434],{},[368,23592,23593,23595],{},[371,23594,7852],{},[355,23596,23597,23599,23601],{},[371,23598,1057],{},[738,23600,5839],{},[361,23602,2883],{},[738,23604,5839],{},[742,23606,23607,23609],{},[371,23608,19142],{},[361,23610,4738],{},[738,23612,23434],{},[368,23614,23615,23617],{},[371,23616,7852],{},[355,23618,23619,23621,23623],{},[371,23620,1057],{},[738,23622,5839],{},[361,23624,4738],{},[738,23626,5839],{},[738,23628,23629],{},"⋯",[738,23631,5839],{},[742,23633,23634,23636],{},[371,23635,19142],{},[371,23637,13358],{},[738,23639,23434],{},[368,23641,23642,23644],{},[371,23643,7852],{},[371,23645,13358],{},[738,23647,18511],{},[368,23649,23650,23652],{},[371,23651,949],{},[371,23653,14101],{},[738,23655,1078],{"stretchy":1077},[368,23657,23658,23660],{},[371,23659,12890],{},[371,23661,1057],{},[738,23663,1092],{"stretchy":1077},[378,23665,23666],{"encoding":380}," A_\\theta^1(s_t, a) = r_t + \\gamma V_\\theta(s_{t+1}) - V_\\theta(s_t)\\\\\n A_\\theta^2(s_t, a) = r_t + \\gamma r_{t+1} + \\gamma^2 V_\\theta(s_{t+2}) - V_\\theta(s_t)\\\\\n A_\\theta^3(s_t, a) = r_t + \\gamma * r_{t+1} + \\gamma^2 * r_{t+2} + \\gamma^3 V_\\theta(s_{t+3}) - V_\\theta(s_t)\\\\\n \\vdots \\\\\n A_\\theta^T(s_t, a) = r_t + \\gamma * r_{t+1} + \\gamma^2 * r_{t+2} + \\gamma^3 * r_{t+3} + \\cdots + \\gamma^T * r_T - V_\\theta(s_t)",[63,23668,23670,23793,23848,23961,24053,24056,24177,24232,24299,24438,24530,24533,24654,24709,24728,24792,24837,24901,25040,25132,25135,25153,25156,25279,25334,25352,25416,25460,25524,25568,25632,25650,25695,25750],{"className":23669,"ariaHidden":386},[385],[63,23671,23673,23677,23729,23732,23772,23775,23778,23781,23784,23787,23790],{"className":23672},[390],[63,23674],{"className":23675,"style":23676},[394],"height:1.1141em;vertical-align:-0.25em;",[63,23678,23680,23683],{"className":23679},[399],[63,23681,1054],{"className":23682},[399,472],[63,23684,23686],{"className":23685},[476],[63,23687,23689,23721],{"className":23688},[411,412],[63,23690,23692,23718],{"className":23691},[416],[63,23693,23696,23707],{"className":23694,"style":23695},[420],"height:0.8641em;",[63,23697,23698,23701],{"style":17457},[63,23699],{"className":23700,"style":817},[428],[63,23702,23704],{"className":23703},[433,434,435,436],[63,23705,14101],{"className":23706,"style":3276},[399,472,436],[63,23708,23709,23712],{"style":16572},[63,23710],{"className":23711,"style":817},[428],[63,23713,23715],{"className":23714},[433,434,435,436],[63,23716,363],{"className":23717},[399,436],[63,23719,507],{"className":23720},[506],[63,23722,23724],{"className":23723},[416],[63,23725,23727],{"className":23726,"style":17487},[420],[63,23728],{},[63,23730,1078],{"className":23731},[403],[63,23733,23735,23738],{"className":23734},[399],[63,23736,12890],{"className":23737},[399,472],[63,23739,23741],{"className":23740},[476],[63,23742,23744,23764],{"className":23743},[411,412],[63,23745,23747,23761],{"className":23746},[416],[63,23748,23750],{"className":23749,"style":12818},[420],[63,23751,23752,23755],{"style":1535},[63,23753],{"className":23754,"style":817},[428],[63,23756,23758],{"className":23757},[433,434,435,436],[63,23759,1057],{"className":23760},[399,472,436],[63,23762,507],{"className":23763},[506],[63,23765,23767],{"className":23766},[416],[63,23768,23770],{"className":23769,"style":1554},[420],[63,23771],{},[63,23773,1083],{"className":23774},[1173],[63,23776],{"className":23777,"style":1177},[776],[63,23779,38],{"className":23780},[399,472],[63,23782,1092],{"className":23783},[593],[63,23785],{"className":23786,"style":777},[776],[63,23788,204],{"className":23789},[781],[63,23791],{"className":23792,"style":777},[776],[63,23794,23796,23799,23839,23842,23845],{"className":23795},[390],[63,23797],{"className":23798,"style":22156},[394],[63,23800,23802,23805],{"className":23801},[399],[63,23803,7852],{"className":23804,"style":3276},[399,472],[63,23806,23808],{"className":23807},[476],[63,23809,23811,23831],{"className":23810},[411,412],[63,23812,23814,23828],{"className":23813},[416],[63,23815,23817],{"className":23816,"style":12818},[420],[63,23818,23819,23822],{"style":13208},[63,23820],{"className":23821,"style":817},[428],[63,23823,23825],{"className":23824},[433,434,435,436],[63,23826,1057],{"className":23827},[399,472,436],[63,23829,507],{"className":23830},[506],[63,23832,23834],{"className":23833},[416],[63,23835,23837],{"className":23836,"style":1554},[420],[63,23838],{},[63,23840],{"className":23841,"style":1920},[776],[63,23843,5839],{"className":23844},[830],[63,23846],{"className":23847,"style":1920},[776],[63,23849,23851,23854,23857,23897,23900,23949,23952,23955,23958],{"className":23850},[390],[63,23852],{"className":23853,"style":1155},[394],[63,23855,19142],{"className":23856,"style":19389},[399,472],[63,23858,23860,23863],{"className":23859},[399],[63,23861,949],{"className":23862,"style":979},[399,472],[63,23864,23866],{"className":23865},[476],[63,23867,23869,23889],{"className":23868},[411,412],[63,23870,23872,23886],{"className":23871},[416],[63,23873,23875],{"className":23874,"style":1532},[420],[63,23876,23877,23880],{"style":20739},[63,23878],{"className":23879,"style":817},[428],[63,23881,23883],{"className":23882},[433,434,435,436],[63,23884,14101],{"className":23885,"style":3276},[399,472,436],[63,23887,507],{"className":23888},[506],[63,23890,23892],{"className":23891},[416],[63,23893,23895],{"className":23894,"style":1554},[420],[63,23896],{},[63,23898,1078],{"className":23899},[403],[63,23901,23903,23906],{"className":23902},[399],[63,23904,12890],{"className":23905},[399,472],[63,23907,23909],{"className":23908},[476],[63,23910,23912,23941],{"className":23911},[411,412],[63,23913,23915,23938],{"className":23914},[416],[63,23916,23918],{"className":23917,"style":3157},[420],[63,23919,23920,23923],{"style":1535},[63,23921],{"className":23922,"style":817},[428],[63,23924,23926],{"className":23925},[433,434,435,436],[63,23927,23929,23932,23935],{"className":23928},[399,436],[63,23930,1057],{"className":23931},[399,472,436],[63,23933,5839],{"className":23934},[830,436],[63,23936,363],{"className":23937},[399,436],[63,23939,507],{"className":23940},[506],[63,23942,23944],{"className":23943},[416],[63,23945,23947],{"className":23946,"style":6959},[420],[63,23948],{},[63,23950,1092],{"className":23951},[593],[63,23953],{"className":23954,"style":1920},[776],[63,23956,18511],{"className":23957},[830],[63,23959],{"className":23960,"style":1920},[776],[63,23962,23964,23967,24007,24010,24050],{"className":23963},[390],[63,23965],{"className":23966,"style":1155},[394],[63,23968,23970,23973],{"className":23969},[399],[63,23971,949],{"className":23972,"style":979},[399,472],[63,23974,23976],{"className":23975},[476],[63,23977,23979,23999],{"className":23978},[411,412],[63,23980,23982,23996],{"className":23981},[416],[63,23983,23985],{"className":23984,"style":1532},[420],[63,23986,23987,23990],{"style":20739},[63,23988],{"className":23989,"style":817},[428],[63,23991,23993],{"className":23992},[433,434,435,436],[63,23994,14101],{"className":23995,"style":3276},[399,472,436],[63,23997,507],{"className":23998},[506],[63,24000,24002],{"className":24001},[416],[63,24003,24005],{"className":24004,"style":1554},[420],[63,24006],{},[63,24008,1078],{"className":24009},[403],[63,24011,24013,24016],{"className":24012},[399],[63,24014,12890],{"className":24015},[399,472],[63,24017,24019],{"className":24018},[476],[63,24020,24022,24042],{"className":24021},[411,412],[63,24023,24025,24039],{"className":24024},[416],[63,24026,24028],{"className":24027,"style":12818},[420],[63,24029,24030,24033],{"style":1535},[63,24031],{"className":24032,"style":817},[428],[63,24034,24036],{"className":24035},[433,434,435,436],[63,24037,1057],{"className":24038},[399,472,436],[63,24040,507],{"className":24041},[506],[63,24043,24045],{"className":24044},[416],[63,24046,24048],{"className":24047,"style":1554},[420],[63,24049],{},[63,24051,1092],{"className":24052},[593],[63,24054],{"className":24055},[776,21897],[63,24057,24059,24062,24113,24116,24156,24159,24162,24165,24168,24171,24174],{"className":24058},[390],[63,24060],{"className":24061,"style":23676},[394],[63,24063,24065,24068],{"className":24064},[399],[63,24066,1054],{"className":24067},[399,472],[63,24069,24071],{"className":24070},[476],[63,24072,24074,24105],{"className":24073},[411,412],[63,24075,24077,24102],{"className":24076},[416],[63,24078,24080,24091],{"className":24079,"style":23695},[420],[63,24081,24082,24085],{"style":17457},[63,24083],{"className":24084,"style":817},[428],[63,24086,24088],{"className":24087},[433,434,435,436],[63,24089,14101],{"className":24090,"style":3276},[399,472,436],[63,24092,24093,24096],{"style":16572},[63,24094],{"className":24095,"style":817},[428],[63,24097,24099],{"className":24098},[433,434,435,436],[63,24100,2883],{"className":24101},[399,436],[63,24103,507],{"className":24104},[506],[63,24106,24108],{"className":24107},[416],[63,24109,24111],{"className":24110,"style":17487},[420],[63,24112],{},[63,24114,1078],{"className":24115},[403],[63,24117,24119,24122],{"className":24118},[399],[63,24120,12890],{"className":24121},[399,472],[63,24123,24125],{"className":24124},[476],[63,24126,24128,24148],{"className":24127},[411,412],[63,24129,24131,24145],{"className":24130},[416],[63,24132,24134],{"className":24133,"style":12818},[420],[63,24135,24136,24139],{"style":1535},[63,24137],{"className":24138,"style":817},[428],[63,24140,24142],{"className":24141},[433,434,435,436],[63,24143,1057],{"className":24144},[399,472,436],[63,24146,507],{"className":24147},[506],[63,24149,24151],{"className":24150},[416],[63,24152,24154],{"className":24153,"style":1554},[420],[63,24155],{},[63,24157,1083],{"className":24158},[1173],[63,24160],{"className":24161,"style":1177},[776],[63,24163,38],{"className":24164},[399,472],[63,24166,1092],{"className":24167},[593],[63,24169],{"className":24170,"style":777},[776],[63,24172,204],{"className":24173},[781],[63,24175],{"className":24176,"style":777},[776],[63,24178,24180,24183,24223,24226,24229],{"className":24179},[390],[63,24181],{"className":24182,"style":22156},[394],[63,24184,24186,24189],{"className":24185},[399],[63,24187,7852],{"className":24188,"style":3276},[399,472],[63,24190,24192],{"className":24191},[476],[63,24193,24195,24215],{"className":24194},[411,412],[63,24196,24198,24212],{"className":24197},[416],[63,24199,24201],{"className":24200,"style":12818},[420],[63,24202,24203,24206],{"style":13208},[63,24204],{"className":24205,"style":817},[428],[63,24207,24209],{"className":24208},[433,434,435,436],[63,24210,1057],{"className":24211},[399,472,436],[63,24213,507],{"className":24214},[506],[63,24216,24218],{"className":24217},[416],[63,24219,24221],{"className":24220,"style":1554},[420],[63,24222],{},[63,24224],{"className":24225,"style":1920},[776],[63,24227,5839],{"className":24228},[830],[63,24230],{"className":24231,"style":1920},[776],[63,24233,24235,24238,24241,24290,24293,24296],{"className":24234},[390],[63,24236],{"className":24237,"style":22802},[394],[63,24239,19142],{"className":24240,"style":19389},[399,472],[63,24242,24244,24247],{"className":24243},[399],[63,24245,7852],{"className":24246,"style":3276},[399,472],[63,24248,24250],{"className":24249},[476],[63,24251,24253,24282],{"className":24252},[411,412],[63,24254,24256,24279],{"className":24255},[416],[63,24257,24259],{"className":24258,"style":3157},[420],[63,24260,24261,24264],{"style":13208},[63,24262],{"className":24263,"style":817},[428],[63,24265,24267],{"className":24266},[433,434,435,436],[63,24268,24270,24273,24276],{"className":24269},[399,436],[63,24271,1057],{"className":24272},[399,472,436],[63,24274,5839],{"className":24275},[830,436],[63,24277,363],{"className":24278},[399,436],[63,24280,507],{"className":24281},[506],[63,24283,24285],{"className":24284},[416],[63,24286,24288],{"className":24287,"style":6959},[420],[63,24289],{},[63,24291],{"className":24292,"style":1920},[776],[63,24294,5839],{"className":24295},[830],[63,24297],{"className":24298,"style":1920},[776],[63,24300,24302,24305,24334,24374,24377,24426,24429,24432,24435],{"className":24301},[390],[63,24303],{"className":24304,"style":23676},[394],[63,24306,24308,24311],{"className":24307},[399],[63,24309,19142],{"className":24310,"style":19389},[399,472],[63,24312,24314],{"className":24313},[476],[63,24315,24317],{"className":24316},[411],[63,24318,24320],{"className":24319},[416],[63,24321,24323],{"className":24322,"style":23695},[420],[63,24324,24325,24328],{"style":16572},[63,24326],{"className":24327,"style":817},[428],[63,24329,24331],{"className":24330},[433,434,435,436],[63,24332,2883],{"className":24333},[399,436],[63,24335,24337,24340],{"className":24336},[399],[63,24338,949],{"className":24339,"style":979},[399,472],[63,24341,24343],{"className":24342},[476],[63,24344,24346,24366],{"className":24345},[411,412],[63,24347,24349,24363],{"className":24348},[416],[63,24350,24352],{"className":24351,"style":1532},[420],[63,24353,24354,24357],{"style":20739},[63,24355],{"className":24356,"style":817},[428],[63,24358,24360],{"className":24359},[433,434,435,436],[63,24361,14101],{"className":24362,"style":3276},[399,472,436],[63,24364,507],{"className":24365},[506],[63,24367,24369],{"className":24368},[416],[63,24370,24372],{"className":24371,"style":1554},[420],[63,24373],{},[63,24375,1078],{"className":24376},[403],[63,24378,24380,24383],{"className":24379},[399],[63,24381,12890],{"className":24382},[399,472],[63,24384,24386],{"className":24385},[476],[63,24387,24389,24418],{"className":24388},[411,412],[63,24390,24392,24415],{"className":24391},[416],[63,24393,24395],{"className":24394,"style":3157},[420],[63,24396,24397,24400],{"style":1535},[63,24398],{"className":24399,"style":817},[428],[63,24401,24403],{"className":24402},[433,434,435,436],[63,24404,24406,24409,24412],{"className":24405},[399,436],[63,24407,1057],{"className":24408},[399,472,436],[63,24410,5839],{"className":24411},[830,436],[63,24413,2883],{"className":24414},[399,436],[63,24416,507],{"className":24417},[506],[63,24419,24421],{"className":24420},[416],[63,24422,24424],{"className":24423,"style":6959},[420],[63,24425],{},[63,24427,1092],{"className":24428},[593],[63,24430],{"className":24431,"style":1920},[776],[63,24433,18511],{"className":24434},[830],[63,24436],{"className":24437,"style":1920},[776],[63,24439,24441,24444,24484,24487,24527],{"className":24440},[390],[63,24442],{"className":24443,"style":1155},[394],[63,24445,24447,24450],{"className":24446},[399],[63,24448,949],{"className":24449,"style":979},[399,472],[63,24451,24453],{"className":24452},[476],[63,24454,24456,24476],{"className":24455},[411,412],[63,24457,24459,24473],{"className":24458},[416],[63,24460,24462],{"className":24461,"style":1532},[420],[63,24463,24464,24467],{"style":20739},[63,24465],{"className":24466,"style":817},[428],[63,24468,24470],{"className":24469},[433,434,435,436],[63,24471,14101],{"className":24472,"style":3276},[399,472,436],[63,24474,507],{"className":24475},[506],[63,24477,24479],{"className":24478},[416],[63,24480,24482],{"className":24481,"style":1554},[420],[63,24483],{},[63,24485,1078],{"className":24486},[403],[63,24488,24490,24493],{"className":24489},[399],[63,24491,12890],{"className":24492},[399,472],[63,24494,24496],{"className":24495},[476],[63,24497,24499,24519],{"className":24498},[411,412],[63,24500,24502,24516],{"className":24501},[416],[63,24503,24505],{"className":24504,"style":12818},[420],[63,24506,24507,24510],{"style":1535},[63,24508],{"className":24509,"style":817},[428],[63,24511,24513],{"className":24512},[433,434,435,436],[63,24514,1057],{"className":24515},[399,472,436],[63,24517,507],{"className":24518},[506],[63,24520,24522],{"className":24521},[416],[63,24523,24525],{"className":24524,"style":1554},[420],[63,24526],{},[63,24528,1092],{"className":24529},[593],[63,24531],{"className":24532},[776,21897],[63,24534,24536,24539,24590,24593,24633,24636,24639,24642,24645,24648,24651],{"className":24535},[390],[63,24537],{"className":24538,"style":23676},[394],[63,24540,24542,24545],{"className":24541},[399],[63,24543,1054],{"className":24544},[399,472],[63,24546,24548],{"className":24547},[476],[63,24549,24551,24582],{"className":24550},[411,412],[63,24552,24554,24579],{"className":24553},[416],[63,24555,24557,24568],{"className":24556,"style":23695},[420],[63,24558,24559,24562],{"style":17457},[63,24560],{"className":24561,"style":817},[428],[63,24563,24565],{"className":24564},[433,434,435,436],[63,24566,14101],{"className":24567,"style":3276},[399,472,436],[63,24569,24570,24573],{"style":16572},[63,24571],{"className":24572,"style":817},[428],[63,24574,24576],{"className":24575},[433,434,435,436],[63,24577,4738],{"className":24578},[399,436],[63,24580,507],{"className":24581},[506],[63,24583,24585],{"className":24584},[416],[63,24586,24588],{"className":24587,"style":17487},[420],[63,24589],{},[63,24591,1078],{"className":24592},[403],[63,24594,24596,24599],{"className":24595},[399],[63,24597,12890],{"className":24598},[399,472],[63,24600,24602],{"className":24601},[476],[63,24603,24605,24625],{"className":24604},[411,412],[63,24606,24608,24622],{"className":24607},[416],[63,24609,24611],{"className":24610,"style":12818},[420],[63,24612,24613,24616],{"style":1535},[63,24614],{"className":24615,"style":817},[428],[63,24617,24619],{"className":24618},[433,434,435,436],[63,24620,1057],{"className":24621},[399,472,436],[63,24623,507],{"className":24624},[506],[63,24626,24628],{"className":24627},[416],[63,24629,24631],{"className":24630,"style":1554},[420],[63,24632],{},[63,24634,1083],{"className":24635},[1173],[63,24637],{"className":24638,"style":1177},[776],[63,24640,38],{"className":24641},[399,472],[63,24643,1092],{"className":24644},[593],[63,24646],{"className":24647,"style":777},[776],[63,24649,204],{"className":24650},[781],[63,24652],{"className":24653,"style":777},[776],[63,24655,24657,24660,24700,24703,24706],{"className":24656},[390],[63,24658],{"className":24659,"style":22156},[394],[63,24661,24663,24666],{"className":24662},[399],[63,24664,7852],{"className":24665,"style":3276},[399,472],[63,24667,24669],{"className":24668},[476],[63,24670,24672,24692],{"className":24671},[411,412],[63,24673,24675,24689],{"className":24674},[416],[63,24676,24678],{"className":24677,"style":12818},[420],[63,24679,24680,24683],{"style":13208},[63,24681],{"className":24682,"style":817},[428],[63,24684,24686],{"className":24685},[433,434,435,436],[63,24687,1057],{"className":24688},[399,472,436],[63,24690,507],{"className":24691},[506],[63,24693,24695],{"className":24694},[416],[63,24696,24698],{"className":24697,"style":1554},[420],[63,24699],{},[63,24701],{"className":24702,"style":1920},[776],[63,24704,5839],{"className":24705},[830],[63,24707],{"className":24708,"style":1920},[776],[63,24710,24712,24716,24719,24722,24725],{"className":24711},[390],[63,24713],{"className":24714,"style":24715},[394],"height:0.6597em;vertical-align:-0.1944em;",[63,24717,19142],{"className":24718,"style":19389},[399,472],[63,24720],{"className":24721,"style":1920},[776],[63,24723,23434],{"className":24724},[830],[63,24726],{"className":24727,"style":1920},[776],[63,24729,24731,24734,24783,24786,24789],{"className":24730},[390],[63,24732],{"className":24733,"style":22802},[394],[63,24735,24737,24740],{"className":24736},[399],[63,24738,7852],{"className":24739,"style":3276},[399,472],[63,24741,24743],{"className":24742},[476],[63,24744,24746,24775],{"className":24745},[411,412],[63,24747,24749,24772],{"className":24748},[416],[63,24750,24752],{"className":24751,"style":3157},[420],[63,24753,24754,24757],{"style":13208},[63,24755],{"className":24756,"style":817},[428],[63,24758,24760],{"className":24759},[433,434,435,436],[63,24761,24763,24766,24769],{"className":24762},[399,436],[63,24764,1057],{"className":24765},[399,472,436],[63,24767,5839],{"className":24768},[830,436],[63,24770,363],{"className":24771},[399,436],[63,24773,507],{"className":24774},[506],[63,24776,24778],{"className":24777},[416],[63,24779,24781],{"className":24780,"style":6959},[420],[63,24782],{},[63,24784],{"className":24785,"style":1920},[776],[63,24787,5839],{"className":24788},[830],[63,24790],{"className":24791,"style":1920},[776],[63,24793,24795,24799,24828,24831,24834],{"className":24794},[390],[63,24796],{"className":24797,"style":24798},[394],"height:1.0585em;vertical-align:-0.1944em;",[63,24800,24802,24805],{"className":24801},[399],[63,24803,19142],{"className":24804,"style":19389},[399,472],[63,24806,24808],{"className":24807},[476],[63,24809,24811],{"className":24810},[411],[63,24812,24814],{"className":24813},[416],[63,24815,24817],{"className":24816,"style":23695},[420],[63,24818,24819,24822],{"style":16572},[63,24820],{"className":24821,"style":817},[428],[63,24823,24825],{"className":24824},[433,434,435,436],[63,24826,2883],{"className":24827},[399,436],[63,24829],{"className":24830,"style":1920},[776],[63,24832,23434],{"className":24833},[830],[63,24835],{"className":24836,"style":1920},[776],[63,24838,24840,24843,24892,24895,24898],{"className":24839},[390],[63,24841],{"className":24842,"style":22802},[394],[63,24844,24846,24849],{"className":24845},[399],[63,24847,7852],{"className":24848,"style":3276},[399,472],[63,24850,24852],{"className":24851},[476],[63,24853,24855,24884],{"className":24854},[411,412],[63,24856,24858,24881],{"className":24857},[416],[63,24859,24861],{"className":24860,"style":3157},[420],[63,24862,24863,24866],{"style":13208},[63,24864],{"className":24865,"style":817},[428],[63,24867,24869],{"className":24868},[433,434,435,436],[63,24870,24872,24875,24878],{"className":24871},[399,436],[63,24873,1057],{"className":24874},[399,472,436],[63,24876,5839],{"className":24877},[830,436],[63,24879,2883],{"className":24880},[399,436],[63,24882,507],{"className":24883},[506],[63,24885,24887],{"className":24886},[416],[63,24888,24890],{"className":24889,"style":6959},[420],[63,24891],{},[63,24893],{"className":24894,"style":1920},[776],[63,24896,5839],{"className":24897},[830],[63,24899],{"className":24900,"style":1920},[776],[63,24902,24904,24907,24936,24976,24979,25028,25031,25034,25037],{"className":24903},[390],[63,24905],{"className":24906,"style":23676},[394],[63,24908,24910,24913],{"className":24909},[399],[63,24911,19142],{"className":24912,"style":19389},[399,472],[63,24914,24916],{"className":24915},[476],[63,24917,24919],{"className":24918},[411],[63,24920,24922],{"className":24921},[416],[63,24923,24925],{"className":24924,"style":23695},[420],[63,24926,24927,24930],{"style":16572},[63,24928],{"className":24929,"style":817},[428],[63,24931,24933],{"className":24932},[433,434,435,436],[63,24934,4738],{"className":24935},[399,436],[63,24937,24939,24942],{"className":24938},[399],[63,24940,949],{"className":24941,"style":979},[399,472],[63,24943,24945],{"className":24944},[476],[63,24946,24948,24968],{"className":24947},[411,412],[63,24949,24951,24965],{"className":24950},[416],[63,24952,24954],{"className":24953,"style":1532},[420],[63,24955,24956,24959],{"style":20739},[63,24957],{"className":24958,"style":817},[428],[63,24960,24962],{"className":24961},[433,434,435,436],[63,24963,14101],{"className":24964,"style":3276},[399,472,436],[63,24966,507],{"className":24967},[506],[63,24969,24971],{"className":24970},[416],[63,24972,24974],{"className":24973,"style":1554},[420],[63,24975],{},[63,24977,1078],{"className":24978},[403],[63,24980,24982,24985],{"className":24981},[399],[63,24983,12890],{"className":24984},[399,472],[63,24986,24988],{"className":24987},[476],[63,24989,24991,25020],{"className":24990},[411,412],[63,24992,24994,25017],{"className":24993},[416],[63,24995,24997],{"className":24996,"style":3157},[420],[63,24998,24999,25002],{"style":1535},[63,25000],{"className":25001,"style":817},[428],[63,25003,25005],{"className":25004},[433,434,435,436],[63,25006,25008,25011,25014],{"className":25007},[399,436],[63,25009,1057],{"className":25010},[399,472,436],[63,25012,5839],{"className":25013},[830,436],[63,25015,4738],{"className":25016},[399,436],[63,25018,507],{"className":25019},[506],[63,25021,25023],{"className":25022},[416],[63,25024,25026],{"className":25025,"style":6959},[420],[63,25027],{},[63,25029,1092],{"className":25030},[593],[63,25032],{"className":25033,"style":1920},[776],[63,25035,18511],{"className":25036},[830],[63,25038],{"className":25039,"style":1920},[776],[63,25041,25043,25046,25086,25089,25129],{"className":25042},[390],[63,25044],{"className":25045,"style":1155},[394],[63,25047,25049,25052],{"className":25048},[399],[63,25050,949],{"className":25051,"style":979},[399,472],[63,25053,25055],{"className":25054},[476],[63,25056,25058,25078],{"className":25057},[411,412],[63,25059,25061,25075],{"className":25060},[416],[63,25062,25064],{"className":25063,"style":1532},[420],[63,25065,25066,25069],{"style":20739},[63,25067],{"className":25068,"style":817},[428],[63,25070,25072],{"className":25071},[433,434,435,436],[63,25073,14101],{"className":25074,"style":3276},[399,472,436],[63,25076,507],{"className":25077},[506],[63,25079,25081],{"className":25080},[416],[63,25082,25084],{"className":25083,"style":1554},[420],[63,25085],{},[63,25087,1078],{"className":25088},[403],[63,25090,25092,25095],{"className":25091},[399],[63,25093,12890],{"className":25094},[399,472],[63,25096,25098],{"className":25097},[476],[63,25099,25101,25121],{"className":25100},[411,412],[63,25102,25104,25118],{"className":25103},[416],[63,25105,25107],{"className":25106,"style":12818},[420],[63,25108,25109,25112],{"style":1535},[63,25110],{"className":25111,"style":817},[428],[63,25113,25115],{"className":25114},[433,434,435,436],[63,25116,1057],{"className":25117},[399,472,436],[63,25119,507],{"className":25120},[506],[63,25122,25124],{"className":25123},[416],[63,25125,25127],{"className":25126,"style":1554},[420],[63,25128],{},[63,25130,1092],{"className":25131},[593],[63,25133],{"className":25134},[776,21897],[63,25136,25138,25142],{"className":25137},[390],[63,25139],{"className":25140,"style":25141},[394],"height:1.53em;vertical-align:-0.03em;",[63,25143,25145,25148],{"className":25144},[399],[63,25146,23523],{"className":25147},[399],[63,25149],{"className":25150,"style":25152},[399,25151],"rule","border-right-width:0em;border-top-width:1.5em;bottom:0em;",[63,25154],{"className":25155},[776,21897],[63,25157,25159,25163,25215,25218,25258,25261,25264,25267,25270,25273,25276],{"className":25158},[390],[63,25160],{"className":25161,"style":25162},[394],"height:1.1413em;vertical-align:-0.25em;",[63,25164,25166,25169],{"className":25165},[399],[63,25167,1054],{"className":25168},[399,472],[63,25170,25172],{"className":25171},[476],[63,25173,25175,25207],{"className":25174},[411,412],[63,25176,25178,25204],{"className":25177},[416],[63,25179,25182,25193],{"className":25180,"style":25181},[420],"height:0.8913em;",[63,25183,25184,25187],{"style":17457},[63,25185],{"className":25186,"style":817},[428],[63,25188,25190],{"className":25189},[433,434,435,436],[63,25191,14101],{"className":25192,"style":3276},[399,472,436],[63,25194,25195,25198],{"style":16572},[63,25196],{"className":25197,"style":817},[428],[63,25199,25201],{"className":25200},[433,434,435,436],[63,25202,13358],{"className":25203,"style":3251},[399,472,436],[63,25205,507],{"className":25206},[506],[63,25208,25210],{"className":25209},[416],[63,25211,25213],{"className":25212,"style":17487},[420],[63,25214],{},[63,25216,1078],{"className":25217},[403],[63,25219,25221,25224],{"className":25220},[399],[63,25222,12890],{"className":25223},[399,472],[63,25225,25227],{"className":25226},[476],[63,25228,25230,25250],{"className":25229},[411,412],[63,25231,25233,25247],{"className":25232},[416],[63,25234,25236],{"className":25235,"style":12818},[420],[63,25237,25238,25241],{"style":1535},[63,25239],{"className":25240,"style":817},[428],[63,25242,25244],{"className":25243},[433,434,435,436],[63,25245,1057],{"className":25246},[399,472,436],[63,25248,507],{"className":25249},[506],[63,25251,25253],{"className":25252},[416],[63,25254,25256],{"className":25255,"style":1554},[420],[63,25257],{},[63,25259,1083],{"className":25260},[1173],[63,25262],{"className":25263,"style":1177},[776],[63,25265,38],{"className":25266},[399,472],[63,25268,1092],{"className":25269},[593],[63,25271],{"className":25272,"style":777},[776],[63,25274,204],{"className":25275},[781],[63,25277],{"className":25278,"style":777},[776],[63,25280,25282,25285,25325,25328,25331],{"className":25281},[390],[63,25283],{"className":25284,"style":22156},[394],[63,25286,25288,25291],{"className":25287},[399],[63,25289,7852],{"className":25290,"style":3276},[399,472],[63,25292,25294],{"className":25293},[476],[63,25295,25297,25317],{"className":25296},[411,412],[63,25298,25300,25314],{"className":25299},[416],[63,25301,25303],{"className":25302,"style":12818},[420],[63,25304,25305,25308],{"style":13208},[63,25306],{"className":25307,"style":817},[428],[63,25309,25311],{"className":25310},[433,434,435,436],[63,25312,1057],{"className":25313},[399,472,436],[63,25315,507],{"className":25316},[506],[63,25318,25320],{"className":25319},[416],[63,25321,25323],{"className":25322,"style":1554},[420],[63,25324],{},[63,25326],{"className":25327,"style":1920},[776],[63,25329,5839],{"className":25330},[830],[63,25332],{"className":25333,"style":1920},[776],[63,25335,25337,25340,25343,25346,25349],{"className":25336},[390],[63,25338],{"className":25339,"style":24715},[394],[63,25341,19142],{"className":25342,"style":19389},[399,472],[63,25344],{"className":25345,"style":1920},[776],[63,25347,23434],{"className":25348},[830],[63,25350],{"className":25351,"style":1920},[776],[63,25353,25355,25358,25407,25410,25413],{"className":25354},[390],[63,25356],{"className":25357,"style":22802},[394],[63,25359,25361,25364],{"className":25360},[399],[63,25362,7852],{"className":25363,"style":3276},[399,472],[63,25365,25367],{"className":25366},[476],[63,25368,25370,25399],{"className":25369},[411,412],[63,25371,25373,25396],{"className":25372},[416],[63,25374,25376],{"className":25375,"style":3157},[420],[63,25377,25378,25381],{"style":13208},[63,25379],{"className":25380,"style":817},[428],[63,25382,25384],{"className":25383},[433,434,435,436],[63,25385,25387,25390,25393],{"className":25386},[399,436],[63,25388,1057],{"className":25389},[399,472,436],[63,25391,5839],{"className":25392},[830,436],[63,25394,363],{"className":25395},[399,436],[63,25397,507],{"className":25398},[506],[63,25400,25402],{"className":25401},[416],[63,25403,25405],{"className":25404,"style":6959},[420],[63,25406],{},[63,25408],{"className":25409,"style":1920},[776],[63,25411,5839],{"className":25412},[830],[63,25414],{"className":25415,"style":1920},[776],[63,25417,25419,25422,25451,25454,25457],{"className":25418},[390],[63,25420],{"className":25421,"style":24798},[394],[63,25423,25425,25428],{"className":25424},[399],[63,25426,19142],{"className":25427,"style":19389},[399,472],[63,25429,25431],{"className":25430},[476],[63,25432,25434],{"className":25433},[411],[63,25435,25437],{"className":25436},[416],[63,25438,25440],{"className":25439,"style":23695},[420],[63,25441,25442,25445],{"style":16572},[63,25443],{"className":25444,"style":817},[428],[63,25446,25448],{"className":25447},[433,434,435,436],[63,25449,2883],{"className":25450},[399,436],[63,25452],{"className":25453,"style":1920},[776],[63,25455,23434],{"className":25456},[830],[63,25458],{"className":25459,"style":1920},[776],[63,25461,25463,25466,25515,25518,25521],{"className":25462},[390],[63,25464],{"className":25465,"style":22802},[394],[63,25467,25469,25472],{"className":25468},[399],[63,25470,7852],{"className":25471,"style":3276},[399,472],[63,25473,25475],{"className":25474},[476],[63,25476,25478,25507],{"className":25477},[411,412],[63,25479,25481,25504],{"className":25480},[416],[63,25482,25484],{"className":25483,"style":3157},[420],[63,25485,25486,25489],{"style":13208},[63,25487],{"className":25488,"style":817},[428],[63,25490,25492],{"className":25491},[433,434,435,436],[63,25493,25495,25498,25501],{"className":25494},[399,436],[63,25496,1057],{"className":25497},[399,472,436],[63,25499,5839],{"className":25500},[830,436],[63,25502,2883],{"className":25503},[399,436],[63,25505,507],{"className":25506},[506],[63,25508,25510],{"className":25509},[416],[63,25511,25513],{"className":25512,"style":6959},[420],[63,25514],{},[63,25516],{"className":25517,"style":1920},[776],[63,25519,5839],{"className":25520},[830],[63,25522],{"className":25523,"style":1920},[776],[63,25525,25527,25530,25559,25562,25565],{"className":25526},[390],[63,25528],{"className":25529,"style":24798},[394],[63,25531,25533,25536],{"className":25532},[399],[63,25534,19142],{"className":25535,"style":19389},[399,472],[63,25537,25539],{"className":25538},[476],[63,25540,25542],{"className":25541},[411],[63,25543,25545],{"className":25544},[416],[63,25546,25548],{"className":25547,"style":23695},[420],[63,25549,25550,25553],{"style":16572},[63,25551],{"className":25552,"style":817},[428],[63,25554,25556],{"className":25555},[433,434,435,436],[63,25557,4738],{"className":25558},[399,436],[63,25560],{"className":25561,"style":1920},[776],[63,25563,23434],{"className":25564},[830],[63,25566],{"className":25567,"style":1920},[776],[63,25569,25571,25574,25623,25626,25629],{"className":25570},[390],[63,25572],{"className":25573,"style":22802},[394],[63,25575,25577,25580],{"className":25576},[399],[63,25578,7852],{"className":25579,"style":3276},[399,472],[63,25581,25583],{"className":25582},[476],[63,25584,25586,25615],{"className":25585},[411,412],[63,25587,25589,25612],{"className":25588},[416],[63,25590,25592],{"className":25591,"style":3157},[420],[63,25593,25594,25597],{"style":13208},[63,25595],{"className":25596,"style":817},[428],[63,25598,25600],{"className":25599},[433,434,435,436],[63,25601,25603,25606,25609],{"className":25602},[399,436],[63,25604,1057],{"className":25605},[399,472,436],[63,25607,5839],{"className":25608},[830,436],[63,25610,4738],{"className":25611},[399,436],[63,25613,507],{"className":25614},[506],[63,25616,25618],{"className":25617},[416],[63,25619,25621],{"className":25620,"style":6959},[420],[63,25622],{},[63,25624],{"className":25625,"style":1920},[776],[63,25627,5839],{"className":25628},[830],[63,25630],{"className":25631,"style":1920},[776],[63,25633,25635,25638,25641,25644,25647],{"className":25634},[390],[63,25636],{"className":25637,"style":8028},[394],[63,25639,23629],{"className":25640},[1222],[63,25642],{"className":25643,"style":1920},[776],[63,25645,5839],{"className":25646},[830],[63,25648],{"className":25649,"style":1920},[776],[63,25651,25653,25657,25686,25689,25692],{"className":25652},[390],[63,25654],{"className":25655,"style":25656},[394],"height:1.0858em;vertical-align:-0.1944em;",[63,25658,25660,25663],{"className":25659},[399],[63,25661,19142],{"className":25662,"style":19389},[399,472],[63,25664,25666],{"className":25665},[476],[63,25667,25669],{"className":25668},[411],[63,25670,25672],{"className":25671},[416],[63,25673,25675],{"className":25674,"style":25181},[420],[63,25676,25677,25680],{"style":16572},[63,25678],{"className":25679,"style":817},[428],[63,25681,25683],{"className":25682},[433,434,435,436],[63,25684,13358],{"className":25685,"style":3251},[399,472,436],[63,25687],{"className":25688,"style":1920},[776],[63,25690,23434],{"className":25691},[830],[63,25693],{"className":25694,"style":1920},[776],[63,25696,25698,25701,25741,25744,25747],{"className":25697},[390],[63,25699],{"className":25700,"style":22156},[394],[63,25702,25704,25707],{"className":25703},[399],[63,25705,7852],{"className":25706,"style":3276},[399,472],[63,25708,25710],{"className":25709},[476],[63,25711,25713,25733],{"className":25712},[411,412],[63,25714,25716,25730],{"className":25715},[416],[63,25717,25719],{"className":25718,"style":13581},[420],[63,25720,25721,25724],{"style":13208},[63,25722],{"className":25723,"style":817},[428],[63,25725,25727],{"className":25726},[433,434,435,436],[63,25728,13358],{"className":25729,"style":3251},[399,472,436],[63,25731,507],{"className":25732},[506],[63,25734,25736],{"className":25735},[416],[63,25737,25739],{"className":25738,"style":1554},[420],[63,25740],{},[63,25742],{"className":25743,"style":1920},[776],[63,25745,18511],{"className":25746},[830],[63,25748],{"className":25749,"style":1920},[776],[63,25751,25753,25756,25796,25799,25839],{"className":25752},[390],[63,25754],{"className":25755,"style":1155},[394],[63,25757,25759,25762],{"className":25758},[399],[63,25760,949],{"className":25761,"style":979},[399,472],[63,25763,25765],{"className":25764},[476],[63,25766,25768,25788],{"className":25767},[411,412],[63,25769,25771,25785],{"className":25770},[416],[63,25772,25774],{"className":25773,"style":1532},[420],[63,25775,25776,25779],{"style":20739},[63,25777],{"className":25778,"style":817},[428],[63,25780,25782],{"className":25781},[433,434,435,436],[63,25783,14101],{"className":25784,"style":3276},[399,472,436],[63,25786,507],{"className":25787},[506],[63,25789,25791],{"className":25790},[416],[63,25792,25794],{"className":25793,"style":1554},[420],[63,25795],{},[63,25797,1078],{"className":25798},[403],[63,25800,25802,25805],{"className":25801},[399],[63,25803,12890],{"className":25804},[399,472],[63,25806,25808],{"className":25807},[476],[63,25809,25811,25831],{"className":25810},[411,412],[63,25812,25814,25828],{"className":25813},[416],[63,25815,25817],{"className":25816,"style":12818},[420],[63,25818,25819,25822],{"style":1535},[63,25820],{"className":25821,"style":817},[428],[63,25823,25825],{"className":25824},[433,434,435,436],[63,25826,1057],{"className":25827},[399,472,436],[63,25829,507],{"className":25830},[506],[63,25832,25834],{"className":25833},[416],[63,25835,25837],{"className":25836,"style":1554},[420],[63,25838],{},[63,25840,1092],{"className":25841},[593],[14,25843,25844,25845,25932,25933,12841],{},"为了让公式更简洁，引入一个中间量 ",[63,25846,25848,25869],{"className":25847},[342],[63,25849,25851],{"className":25850},[346],[348,25852,25853],{"xmlns":350},[352,25854,25855,25866],{},[355,25856,25857],{},[1870,25858,25859,25862,25864],{},[371,25860,25861],{},"δ",[371,25863,1057],{},[371,25865,949],{},[378,25867,25868],{"encoding":380},"\\delta_t^V",[63,25870,25872],{"className":25871,"ariaHidden":386},[385],[63,25873,25875,25879],{"className":25874},[390],[63,25876],{"className":25877,"style":25878},[394],"height:1.0883em;vertical-align:-0.247em;",[63,25880,25882,25886],{"className":25881},[399],[63,25883,25861],{"className":25884,"style":25885},[399,472],"margin-right:0.03785em;",[63,25887,25889],{"className":25888},[476],[63,25890,25892,25924],{"className":25891},[411,412],[63,25893,25895,25921],{"className":25894},[416],[63,25896,25898,25910],{"className":25897,"style":3264},[420],[63,25899,25901,25904],{"style":25900},"top:-2.453em;margin-left:-0.0379em;margin-right:0.05em;",[63,25902],{"className":25903,"style":817},[428],[63,25905,25907],{"className":25906},[433,434,435,436],[63,25908,1057],{"className":25909},[399,472,436],[63,25911,25912,25915],{"style":813},[63,25913],{"className":25914,"style":817},[428],[63,25916,25918],{"className":25917},[433,434,435,436],[63,25919,949],{"className":25920,"style":979},[399,472,436],[63,25922,507],{"className":25923},[506],[63,25925,25927],{"className":25926},[416],[63,25928,25930],{"className":25929,"style":17487},[420],[63,25931],{},"，表示 ",[11217,25934,25935,25936,25964],{},"第 ",[63,25937,25939,25952],{"className":25938},[342],[63,25940,25942],{"className":25941},[346],[348,25943,25944],{"xmlns":350},[352,25945,25946,25950],{},[355,25947,25948],{},[371,25949,1057],{},[378,25951,1057],{"encoding":380},[63,25953,25955],{"className":25954,"ariaHidden":386},[385],[63,25956,25958,25961],{"className":25957},[390],[63,25959],{"className":25960,"style":12867},[394],[63,25962,1057],{"className":25963},[399,472]," 步采取该动作所带来的 TD 残差（temporal-difference error）",[63,25966,25968],{"className":25967},[14495],[63,25969,25971,26297],{"className":25970},[342],[63,25972,25974],{"className":25973},[346],[348,25975,25976],{"xmlns":350,"display":9561},[352,25977,25978,26294],{},[355,25979,25980,25988,25990,25996,25998,26000,26002,26008,26010,26022,26024,26026,26032,26034,26040,26042,26044,26058,26060,26072,26074,26076,26078,26084,26086,26098,26100,26102,26108,26110,26122,26124,26126,26134,26136,26142,26144,26146,26148,26150,26158,26160,26168,26170,26176,26178,26180,26182,26184,26192,26194,26196,26210,26212,26220,26222,26228,26230,26232,26234,26236,26244,26246,26248,26262,26264,26270,26284,26286],{},[1870,25981,25982,25984,25986],{},[371,25983,25861],{},[371,25985,1057],{},[371,25987,949],{},[738,25989,204],{},[368,25991,25992,25994],{},[371,25993,7852],{},[371,25995,1057],{},[738,25997,5839],{},[371,25999,19142],{},[738,26001,23434],{},[368,26003,26004,26006],{},[371,26005,949],{},[371,26007,14101],{},[738,26009,1078],{"stretchy":1077},[368,26011,26012,26014],{},[371,26013,12890],{},[355,26015,26016,26018,26020],{},[371,26017,1057],{},[738,26019,5839],{},[361,26021,363],{},[738,26023,1092],{"stretchy":1077},[738,26025,18511],{},[368,26027,26028,26030],{},[371,26029,949],{},[371,26031,14101],{},[738,26033,1078],{"stretchy":1077},[368,26035,26036,26038],{},[371,26037,12890],{},[371,26039,1057],{},[738,26041,1092],{"stretchy":1077},[776,26043],{"linebreak":21897},[1870,26045,26046,26048,26056],{},[371,26047,25861],{},[355,26049,26050,26052,26054],{},[371,26051,1057],{},[738,26053,5839],{},[361,26055,363],{},[371,26057,949],{},[738,26059,204],{},[368,26061,26062,26064],{},[371,26063,7852],{},[355,26065,26066,26068,26070],{},[371,26067,1057],{},[738,26069,5839],{},[361,26071,363],{},[738,26073,5839],{},[371,26075,19142],{},[738,26077,23434],{},[368,26079,26080,26082],{},[371,26081,949],{},[371,26083,14101],{},[738,26085,1078],{"stretchy":1077},[368,26087,26088,26090],{},[371,26089,12890],{},[355,26091,26092,26094,26096],{},[371,26093,1057],{},[738,26095,5839],{},[361,26097,2883],{},[738,26099,1092],{"stretchy":1077},[738,26101,18511],{},[368,26103,26104,26106],{},[371,26105,949],{},[371,26107,14101],{},[738,26109,1078],{"stretchy":1077},[368,26111,26112,26114],{},[371,26113,12890],{},[355,26115,26116,26118,26120],{},[371,26117,1057],{},[738,26119,5839],{},[361,26121,363],{},[738,26123,1092],{"stretchy":1077},[776,26125],{"linebreak":21897},[1870,26127,26128,26130,26132],{},[371,26129,1054],{},[371,26131,14101],{},[361,26133,363],{},[738,26135,1078],{"stretchy":1077},[368,26137,26138,26140],{},[371,26139,12890],{},[371,26141,1057],{},[738,26143,1083],{"separator":386},[371,26145,38],{},[738,26147,1092],{"stretchy":1077},[738,26149,204],{},[1870,26151,26152,26154,26156],{},[371,26153,25861],{},[371,26155,1057],{},[371,26157,949],{},[776,26159],{"linebreak":21897},[1870,26161,26162,26164,26166],{},[371,26163,1054],{},[371,26165,14101],{},[361,26167,2883],{},[738,26169,1078],{"stretchy":1077},[368,26171,26172,26174],{},[371,26173,12890],{},[371,26175,1057],{},[738,26177,1083],{"separator":386},[371,26179,38],{},[738,26181,1092],{"stretchy":1077},[738,26183,204],{},[1870,26185,26186,26188,26190],{},[371,26187,25861],{},[371,26189,1057],{},[371,26191,949],{},[738,26193,5839],{},[371,26195,19142],{},[1870,26197,26198,26200,26208],{},[371,26199,25861],{},[355,26201,26202,26204,26206],{},[371,26203,1057],{},[738,26205,5839],{},[361,26207,363],{},[371,26209,949],{},[776,26211],{"linebreak":21897},[1870,26213,26214,26216,26218],{},[371,26215,1054],{},[371,26217,14101],{},[361,26219,4738],{},[738,26221,1078],{"stretchy":1077},[368,26223,26224,26226],{},[371,26225,12890],{},[371,26227,1057],{},[738,26229,1083],{"separator":386},[371,26231,38],{},[738,26233,1092],{"stretchy":1077},[738,26235,204],{},[1870,26237,26238,26240,26242],{},[371,26239,25861],{},[371,26241,1057],{},[371,26243,949],{},[738,26245,5839],{},[371,26247,19142],{},[1870,26249,26250,26252,26260],{},[371,26251,25861],{},[355,26253,26254,26256,26258],{},[371,26255,1057],{},[738,26257,5839],{},[361,26259,363],{},[371,26261,949],{},[738,26263,5839],{},[742,26265,26266,26268],{},[371,26267,19142],{},[361,26269,2883],{},[1870,26271,26272,26274,26282],{},[371,26273,25861],{},[355,26275,26276,26278,26280],{},[371,26277,1057],{},[738,26279,5839],{},[361,26281,2883],{},[371,26283,949],{},[776,26285],{"linebreak":21897},[355,26287,26288,26290],{},[371,26289,23523],{"mathvariant":1053},[23525,26291,26292],{"height":14511,"voffset":14511},[776,26293],{"mathbackground":23529,"width":14511,"height":23530},[378,26295,26296],{"encoding":380},"\n\\delta_t^V = r_t + \\gamma * V_\\theta(s_{t+1}) - V_\\theta(s_t)\\\\\n\n\\delta_{t+1}^V = r_{t+1} + \\gamma * V_\\theta(s_{t+2}) - V_\\theta(s_{t+1})\\\\\n\nA_\\theta^1(s_t, a) = \\delta_t^V\\\\\n\nA_\\theta^2(s_t, a) = \\delta_t^V + \\gamma \\delta_{t+1}^V\\\\\n\nA_\\theta^3(s_t, a) = \\delta_t^V + \\gamma \\delta_{t+1}^V + \\gamma^2 \\delta_{t+2}^V\\\\\n\n\\vdots",[63,26298,26300,26367,26422,26440,26550,26642,26645,26722,26786,26804,26914,27015,27018,27139,27196,27199,27320,27386,27455,27458,27579,27645,27723,27818,27821],{"className":26299,"ariaHidden":386},[385],[63,26301,26303,26307,26358,26361,26364],{"className":26302},[390],[63,26304],{"className":26305,"style":26306},[394],"height:1.1383em;vertical-align:-0.247em;",[63,26308,26310,26313],{"className":26309},[399],[63,26311,25861],{"className":26312,"style":25885},[399,472],[63,26314,26316],{"className":26315},[476],[63,26317,26319,26350],{"className":26318},[411,412],[63,26320,26322,26347],{"className":26321},[416],[63,26323,26325,26336],{"className":26324,"style":25181},[420],[63,26326,26327,26330],{"style":25900},[63,26328],{"className":26329,"style":817},[428],[63,26331,26333],{"className":26332},[433,434,435,436],[63,26334,1057],{"className":26335},[399,472,436],[63,26337,26338,26341],{"style":16572},[63,26339],{"className":26340,"style":817},[428],[63,26342,26344],{"className":26343},[433,434,435,436],[63,26345,949],{"className":26346,"style":979},[399,472,436],[63,26348,507],{"className":26349},[506],[63,26351,26353],{"className":26352},[416],[63,26354,26356],{"className":26355,"style":17487},[420],[63,26357],{},[63,26359],{"className":26360,"style":777},[776],[63,26362,204],{"className":26363},[781],[63,26365],{"className":26366,"style":777},[776],[63,26368,26370,26373,26413,26416,26419],{"className":26369},[390],[63,26371],{"className":26372,"style":22156},[394],[63,26374,26376,26379],{"className":26375},[399],[63,26377,7852],{"className":26378,"style":3276},[399,472],[63,26380,26382],{"className":26381},[476],[63,26383,26385,26405],{"className":26384},[411,412],[63,26386,26388,26402],{"className":26387},[416],[63,26389,26391],{"className":26390,"style":12818},[420],[63,26392,26393,26396],{"style":13208},[63,26394],{"className":26395,"style":817},[428],[63,26397,26399],{"className":26398},[433,434,435,436],[63,26400,1057],{"className":26401},[399,472,436],[63,26403,507],{"className":26404},[506],[63,26406,26408],{"className":26407},[416],[63,26409,26411],{"className":26410,"style":1554},[420],[63,26412],{},[63,26414],{"className":26415,"style":1920},[776],[63,26417,5839],{"className":26418},[830],[63,26420],{"className":26421,"style":1920},[776],[63,26423,26425,26428,26431,26434,26437],{"className":26424},[390],[63,26426],{"className":26427,"style":24715},[394],[63,26429,19142],{"className":26430,"style":19389},[399,472],[63,26432],{"className":26433,"style":1920},[776],[63,26435,23434],{"className":26436},[830],[63,26438],{"className":26439,"style":1920},[776],[63,26441,26443,26446,26486,26489,26538,26541,26544,26547],{"className":26442},[390],[63,26444],{"className":26445,"style":1155},[394],[63,26447,26449,26452],{"className":26448},[399],[63,26450,949],{"className":26451,"style":979},[399,472],[63,26453,26455],{"className":26454},[476],[63,26456,26458,26478],{"className":26457},[411,412],[63,26459,26461,26475],{"className":26460},[416],[63,26462,26464],{"className":26463,"style":1532},[420],[63,26465,26466,26469],{"style":20739},[63,26467],{"className":26468,"style":817},[428],[63,26470,26472],{"className":26471},[433,434,435,436],[63,26473,14101],{"className":26474,"style":3276},[399,472,436],[63,26476,507],{"className":26477},[506],[63,26479,26481],{"className":26480},[416],[63,26482,26484],{"className":26483,"style":1554},[420],[63,26485],{},[63,26487,1078],{"className":26488},[403],[63,26490,26492,26495],{"className":26491},[399],[63,26493,12890],{"className":26494},[399,472],[63,26496,26498],{"className":26497},[476],[63,26499,26501,26530],{"className":26500},[411,412],[63,26502,26504,26527],{"className":26503},[416],[63,26505,26507],{"className":26506,"style":3157},[420],[63,26508,26509,26512],{"style":1535},[63,26510],{"className":26511,"style":817},[428],[63,26513,26515],{"className":26514},[433,434,435,436],[63,26516,26518,26521,26524],{"className":26517},[399,436],[63,26519,1057],{"className":26520},[399,472,436],[63,26522,5839],{"className":26523},[830,436],[63,26525,363],{"className":26526},[399,436],[63,26528,507],{"className":26529},[506],[63,26531,26533],{"className":26532},[416],[63,26534,26536],{"className":26535,"style":6959},[420],[63,26537],{},[63,26539,1092],{"className":26540},[593],[63,26542],{"className":26543,"style":1920},[776],[63,26545,18511],{"className":26546},[830],[63,26548],{"className":26549,"style":1920},[776],[63,26551,26553,26556,26596,26599,26639],{"className":26552},[390],[63,26554],{"className":26555,"style":1155},[394],[63,26557,26559,26562],{"className":26558},[399],[63,26560,949],{"className":26561,"style":979},[399,472],[63,26563,26565],{"className":26564},[476],[63,26566,26568,26588],{"className":26567},[411,412],[63,26569,26571,26585],{"className":26570},[416],[63,26572,26574],{"className":26573,"style":1532},[420],[63,26575,26576,26579],{"style":20739},[63,26577],{"className":26578,"style":817},[428],[63,26580,26582],{"className":26581},[433,434,435,436],[63,26583,14101],{"className":26584,"style":3276},[399,472,436],[63,26586,507],{"className":26587},[506],[63,26589,26591],{"className":26590},[416],[63,26592,26594],{"className":26593,"style":1554},[420],[63,26595],{},[63,26597,1078],{"className":26598},[403],[63,26600,26602,26605],{"className":26601},[399],[63,26603,12890],{"className":26604},[399,472],[63,26606,26608],{"className":26607},[476],[63,26609,26611,26631],{"className":26610},[411,412],[63,26612,26614,26628],{"className":26613},[416],[63,26615,26617],{"className":26616,"style":12818},[420],[63,26618,26619,26622],{"style":1535},[63,26620],{"className":26621,"style":817},[428],[63,26623,26625],{"className":26624},[433,434,435,436],[63,26626,1057],{"className":26627},[399,472,436],[63,26629,507],{"className":26630},[506],[63,26632,26634],{"className":26633},[416],[63,26635,26637],{"className":26636,"style":1554},[420],[63,26638],{},[63,26640,1092],{"className":26641},[593],[63,26643],{"className":26644},[776,21897],[63,26646,26648,26652,26713,26716,26719],{"className":26647},[390],[63,26649],{"className":26650,"style":26651},[394],"height:1.1967em;vertical-align:-0.3053em;",[63,26653,26655,26658],{"className":26654},[399],[63,26656,25861],{"className":26657,"style":25885},[399,472],[63,26659,26661],{"className":26660},[476],[63,26662,26664,26704],{"className":26663},[411,412],[63,26665,26667,26701],{"className":26666},[416],[63,26668,26670,26690],{"className":26669,"style":25181},[420],[63,26671,26672,26675],{"style":25900},[63,26673],{"className":26674,"style":817},[428],[63,26676,26678],{"className":26677},[433,434,435,436],[63,26679,26681,26684,26687],{"className":26680},[399,436],[63,26682,1057],{"className":26683},[399,472,436],[63,26685,5839],{"className":26686},[830,436],[63,26688,363],{"className":26689},[399,436],[63,26691,26692,26695],{"style":16572},[63,26693],{"className":26694,"style":817},[428],[63,26696,26698],{"className":26697},[433,434,435,436],[63,26699,949],{"className":26700,"style":979},[399,472,436],[63,26702,507],{"className":26703},[506],[63,26705,26707],{"className":26706},[416],[63,26708,26711],{"className":26709,"style":26710},[420],"height:0.3053em;",[63,26712],{},[63,26714],{"className":26715,"style":777},[776],[63,26717,204],{"className":26718},[781],[63,26720],{"className":26721,"style":777},[776],[63,26723,26725,26728,26777,26780,26783],{"className":26724},[390],[63,26726],{"className":26727,"style":22802},[394],[63,26729,26731,26734],{"className":26730},[399],[63,26732,7852],{"className":26733,"style":3276},[399,472],[63,26735,26737],{"className":26736},[476],[63,26738,26740,26769],{"className":26739},[411,412],[63,26741,26743,26766],{"className":26742},[416],[63,26744,26746],{"className":26745,"style":3157},[420],[63,26747,26748,26751],{"style":13208},[63,26749],{"className":26750,"style":817},[428],[63,26752,26754],{"className":26753},[433,434,435,436],[63,26755,26757,26760,26763],{"className":26756},[399,436],[63,26758,1057],{"className":26759},[399,472,436],[63,26761,5839],{"className":26762},[830,436],[63,26764,363],{"className":26765},[399,436],[63,26767,507],{"className":26768},[506],[63,26770,26772],{"className":26771},[416],[63,26773,26775],{"className":26774,"style":6959},[420],[63,26776],{},[63,26778],{"className":26779,"style":1920},[776],[63,26781,5839],{"className":26782},[830],[63,26784],{"className":26785,"style":1920},[776],[63,26787,26789,26792,26795,26798,26801],{"className":26788},[390],[63,26790],{"className":26791,"style":24715},[394],[63,26793,19142],{"className":26794,"style":19389},[399,472],[63,26796],{"className":26797,"style":1920},[776],[63,26799,23434],{"className":26800},[830],[63,26802],{"className":26803,"style":1920},[776],[63,26805,26807,26810,26850,26853,26902,26905,26908,26911],{"className":26806},[390],[63,26808],{"className":26809,"style":1155},[394],[63,26811,26813,26816],{"className":26812},[399],[63,26814,949],{"className":26815,"style":979},[399,472],[63,26817,26819],{"className":26818},[476],[63,26820,26822,26842],{"className":26821},[411,412],[63,26823,26825,26839],{"className":26824},[416],[63,26826,26828],{"className":26827,"style":1532},[420],[63,26829,26830,26833],{"style":20739},[63,26831],{"className":26832,"style":817},[428],[63,26834,26836],{"className":26835},[433,434,435,436],[63,26837,14101],{"className":26838,"style":3276},[399,472,436],[63,26840,507],{"className":26841},[506],[63,26843,26845],{"className":26844},[416],[63,26846,26848],{"className":26847,"style":1554},[420],[63,26849],{},[63,26851,1078],{"className":26852},[403],[63,26854,26856,26859],{"className":26855},[399],[63,26857,12890],{"className":26858},[399,472],[63,26860,26862],{"className":26861},[476],[63,26863,26865,26894],{"className":26864},[411,412],[63,26866,26868,26891],{"className":26867},[416],[63,26869,26871],{"className":26870,"style":3157},[420],[63,26872,26873,26876],{"style":1535},[63,26874],{"className":26875,"style":817},[428],[63,26877,26879],{"className":26878},[433,434,435,436],[63,26880,26882,26885,26888],{"className":26881},[399,436],[63,26883,1057],{"className":26884},[399,472,436],[63,26886,5839],{"className":26887},[830,436],[63,26889,2883],{"className":26890},[399,436],[63,26892,507],{"className":26893},[506],[63,26895,26897],{"className":26896},[416],[63,26898,26900],{"className":26899,"style":6959},[420],[63,26901],{},[63,26903,1092],{"className":26904},[593],[63,26906],{"className":26907,"style":1920},[776],[63,26909,18511],{"className":26910},[830],[63,26912],{"className":26913,"style":1920},[776],[63,26915,26917,26920,26960,26963,27012],{"className":26916},[390],[63,26918],{"className":26919,"style":1155},[394],[63,26921,26923,26926],{"className":26922},[399],[63,26924,949],{"className":26925,"style":979},[399,472],[63,26927,26929],{"className":26928},[476],[63,26930,26932,26952],{"className":26931},[411,412],[63,26933,26935,26949],{"className":26934},[416],[63,26936,26938],{"className":26937,"style":1532},[420],[63,26939,26940,26943],{"style":20739},[63,26941],{"className":26942,"style":817},[428],[63,26944,26946],{"className":26945},[433,434,435,436],[63,26947,14101],{"className":26948,"style":3276},[399,472,436],[63,26950,507],{"className":26951},[506],[63,26953,26955],{"className":26954},[416],[63,26956,26958],{"className":26957,"style":1554},[420],[63,26959],{},[63,26961,1078],{"className":26962},[403],[63,26964,26966,26969],{"className":26965},[399],[63,26967,12890],{"className":26968},[399,472],[63,26970,26972],{"className":26971},[476],[63,26973,26975,27004],{"className":26974},[411,412],[63,26976,26978,27001],{"className":26977},[416],[63,26979,26981],{"className":26980,"style":3157},[420],[63,26982,26983,26986],{"style":1535},[63,26984],{"className":26985,"style":817},[428],[63,26987,26989],{"className":26988},[433,434,435,436],[63,26990,26992,26995,26998],{"className":26991},[399,436],[63,26993,1057],{"className":26994},[399,472,436],[63,26996,5839],{"className":26997},[830,436],[63,26999,363],{"className":27000},[399,436],[63,27002,507],{"className":27003},[506],[63,27005,27007],{"className":27006},[416],[63,27008,27010],{"className":27009,"style":6959},[420],[63,27011],{},[63,27013,1092],{"className":27014},[593],[63,27016],{"className":27017},[776,21897],[63,27019,27021,27024,27075,27078,27118,27121,27124,27127,27130,27133,27136],{"className":27020},[390],[63,27022],{"className":27023,"style":23676},[394],[63,27025,27027,27030],{"className":27026},[399],[63,27028,1054],{"className":27029},[399,472],[63,27031,27033],{"className":27032},[476],[63,27034,27036,27067],{"className":27035},[411,412],[63,27037,27039,27064],{"className":27038},[416],[63,27040,27042,27053],{"className":27041,"style":23695},[420],[63,27043,27044,27047],{"style":17457},[63,27045],{"className":27046,"style":817},[428],[63,27048,27050],{"className":27049},[433,434,435,436],[63,27051,14101],{"className":27052,"style":3276},[399,472,436],[63,27054,27055,27058],{"style":16572},[63,27056],{"className":27057,"style":817},[428],[63,27059,27061],{"className":27060},[433,434,435,436],[63,27062,363],{"className":27063},[399,436],[63,27065,507],{"className":27066},[506],[63,27068,27070],{"className":27069},[416],[63,27071,27073],{"className":27072,"style":17487},[420],[63,27074],{},[63,27076,1078],{"className":27077},[403],[63,27079,27081,27084],{"className":27080},[399],[63,27082,12890],{"className":27083},[399,472],[63,27085,27087],{"className":27086},[476],[63,27088,27090,27110],{"className":27089},[411,412],[63,27091,27093,27107],{"className":27092},[416],[63,27094,27096],{"className":27095,"style":12818},[420],[63,27097,27098,27101],{"style":1535},[63,27099],{"className":27100,"style":817},[428],[63,27102,27104],{"className":27103},[433,434,435,436],[63,27105,1057],{"className":27106},[399,472,436],[63,27108,507],{"className":27109},[506],[63,27111,27113],{"className":27112},[416],[63,27114,27116],{"className":27115,"style":1554},[420],[63,27117],{},[63,27119,1083],{"className":27120},[1173],[63,27122],{"className":27123,"style":1177},[776],[63,27125,38],{"className":27126},[399,472],[63,27128,1092],{"className":27129},[593],[63,27131],{"className":27132,"style":777},[776],[63,27134,204],{"className":27135},[781],[63,27137],{"className":27138,"style":777},[776],[63,27140,27142,27145],{"className":27141},[390],[63,27143],{"className":27144,"style":26306},[394],[63,27146,27148,27151],{"className":27147},[399],[63,27149,25861],{"className":27150,"style":25885},[399,472],[63,27152,27154],{"className":27153},[476],[63,27155,27157,27188],{"className":27156},[411,412],[63,27158,27160,27185],{"className":27159},[416],[63,27161,27163,27174],{"className":27162,"style":25181},[420],[63,27164,27165,27168],{"style":25900},[63,27166],{"className":27167,"style":817},[428],[63,27169,27171],{"className":27170},[433,434,435,436],[63,27172,1057],{"className":27173},[399,472,436],[63,27175,27176,27179],{"style":16572},[63,27177],{"className":27178,"style":817},[428],[63,27180,27182],{"className":27181},[433,434,435,436],[63,27183,949],{"className":27184,"style":979},[399,472,436],[63,27186,507],{"className":27187},[506],[63,27189,27191],{"className":27190},[416],[63,27192,27194],{"className":27193,"style":17487},[420],[63,27195],{},[63,27197],{"className":27198},[776,21897],[63,27200,27202,27205,27256,27259,27299,27302,27305,27308,27311,27314,27317],{"className":27201},[390],[63,27203],{"className":27204,"style":23676},[394],[63,27206,27208,27211],{"className":27207},[399],[63,27209,1054],{"className":27210},[399,472],[63,27212,27214],{"className":27213},[476],[63,27215,27217,27248],{"className":27216},[411,412],[63,27218,27220,27245],{"className":27219},[416],[63,27221,27223,27234],{"className":27222,"style":23695},[420],[63,27224,27225,27228],{"style":17457},[63,27226],{"className":27227,"style":817},[428],[63,27229,27231],{"className":27230},[433,434,435,436],[63,27232,14101],{"className":27233,"style":3276},[399,472,436],[63,27235,27236,27239],{"style":16572},[63,27237],{"className":27238,"style":817},[428],[63,27240,27242],{"className":27241},[433,434,435,436],[63,27243,2883],{"className":27244},[399,436],[63,27246,507],{"className":27247},[506],[63,27249,27251],{"className":27250},[416],[63,27252,27254],{"className":27253,"style":17487},[420],[63,27255],{},[63,27257,1078],{"className":27258},[403],[63,27260,27262,27265],{"className":27261},[399],[63,27263,12890],{"className":27264},[399,472],[63,27266,27268],{"className":27267},[476],[63,27269,27271,27291],{"className":27270},[411,412],[63,27272,27274,27288],{"className":27273},[416],[63,27275,27277],{"className":27276,"style":12818},[420],[63,27278,27279,27282],{"style":1535},[63,27280],{"className":27281,"style":817},[428],[63,27283,27285],{"className":27284},[433,434,435,436],[63,27286,1057],{"className":27287},[399,472,436],[63,27289,507],{"className":27290},[506],[63,27292,27294],{"className":27293},[416],[63,27295,27297],{"className":27296,"style":1554},[420],[63,27298],{},[63,27300,1083],{"className":27301},[1173],[63,27303],{"className":27304,"style":1177},[776],[63,27306,38],{"className":27307},[399,472],[63,27309,1092],{"className":27310},[593],[63,27312],{"className":27313,"style":777},[776],[63,27315,204],{"className":27316},[781],[63,27318],{"className":27319,"style":777},[776],[63,27321,27323,27326,27377,27380,27383],{"className":27322},[390],[63,27324],{"className":27325,"style":26306},[394],[63,27327,27329,27332],{"className":27328},[399],[63,27330,25861],{"className":27331,"style":25885},[399,472],[63,27333,27335],{"className":27334},[476],[63,27336,27338,27369],{"className":27337},[411,412],[63,27339,27341,27366],{"className":27340},[416],[63,27342,27344,27355],{"className":27343,"style":25181},[420],[63,27345,27346,27349],{"style":25900},[63,27347],{"className":27348,"style":817},[428],[63,27350,27352],{"className":27351},[433,434,435,436],[63,27353,1057],{"className":27354},[399,472,436],[63,27356,27357,27360],{"style":16572},[63,27358],{"className":27359,"style":817},[428],[63,27361,27363],{"className":27362},[433,434,435,436],[63,27364,949],{"className":27365,"style":979},[399,472,436],[63,27367,507],{"className":27368},[506],[63,27370,27372],{"className":27371},[416],[63,27373,27375],{"className":27374,"style":17487},[420],[63,27376],{},[63,27378],{"className":27379,"style":1920},[776],[63,27381,5839],{"className":27382},[830],[63,27384],{"className":27385,"style":1920},[776],[63,27387,27389,27392,27395],{"className":27388},[390],[63,27390],{"className":27391,"style":26651},[394],[63,27393,19142],{"className":27394,"style":19389},[399,472],[63,27396,27398,27401],{"className":27397},[399],[63,27399,25861],{"className":27400,"style":25885},[399,472],[63,27402,27404],{"className":27403},[476],[63,27405,27407,27447],{"className":27406},[411,412],[63,27408,27410,27444],{"className":27409},[416],[63,27411,27413,27433],{"className":27412,"style":25181},[420],[63,27414,27415,27418],{"style":25900},[63,27416],{"className":27417,"style":817},[428],[63,27419,27421],{"className":27420},[433,434,435,436],[63,27422,27424,27427,27430],{"className":27423},[399,436],[63,27425,1057],{"className":27426},[399,472,436],[63,27428,5839],{"className":27429},[830,436],[63,27431,363],{"className":27432},[399,436],[63,27434,27435,27438],{"style":16572},[63,27436],{"className":27437,"style":817},[428],[63,27439,27441],{"className":27440},[433,434,435,436],[63,27442,949],{"className":27443,"style":979},[399,472,436],[63,27445,507],{"className":27446},[506],[63,27448,27450],{"className":27449},[416],[63,27451,27453],{"className":27452,"style":26710},[420],[63,27454],{},[63,27456],{"className":27457},[776,21897],[63,27459,27461,27464,27515,27518,27558,27561,27564,27567,27570,27573,27576],{"className":27460},[390],[63,27462],{"className":27463,"style":23676},[394],[63,27465,27467,27470],{"className":27466},[399],[63,27468,1054],{"className":27469},[399,472],[63,27471,27473],{"className":27472},[476],[63,27474,27476,27507],{"className":27475},[411,412],[63,27477,27479,27504],{"className":27478},[416],[63,27480,27482,27493],{"className":27481,"style":23695},[420],[63,27483,27484,27487],{"style":17457},[63,27485],{"className":27486,"style":817},[428],[63,27488,27490],{"className":27489},[433,434,435,436],[63,27491,14101],{"className":27492,"style":3276},[399,472,436],[63,27494,27495,27498],{"style":16572},[63,27496],{"className":27497,"style":817},[428],[63,27499,27501],{"className":27500},[433,434,435,436],[63,27502,4738],{"className":27503},[399,436],[63,27505,507],{"className":27506},[506],[63,27508,27510],{"className":27509},[416],[63,27511,27513],{"className":27512,"style":17487},[420],[63,27514],{},[63,27516,1078],{"className":27517},[403],[63,27519,27521,27524],{"className":27520},[399],[63,27522,12890],{"className":27523},[399,472],[63,27525,27527],{"className":27526},[476],[63,27528,27530,27550],{"className":27529},[411,412],[63,27531,27533,27547],{"className":27532},[416],[63,27534,27536],{"className":27535,"style":12818},[420],[63,27537,27538,27541],{"style":1535},[63,27539],{"className":27540,"style":817},[428],[63,27542,27544],{"className":27543},[433,434,435,436],[63,27545,1057],{"className":27546},[399,472,436],[63,27548,507],{"className":27549},[506],[63,27551,27553],{"className":27552},[416],[63,27554,27556],{"className":27555,"style":1554},[420],[63,27557],{},[63,27559,1083],{"className":27560},[1173],[63,27562],{"className":27563,"style":1177},[776],[63,27565,38],{"className":27566},[399,472],[63,27568,1092],{"className":27569},[593],[63,27571],{"className":27572,"style":777},[776],[63,27574,204],{"className":27575},[781],[63,27577],{"className":27578,"style":777},[776],[63,27580,27582,27585,27636,27639,27642],{"className":27581},[390],[63,27583],{"className":27584,"style":26306},[394],[63,27586,27588,27591],{"className":27587},[399],[63,27589,25861],{"className":27590,"style":25885},[399,472],[63,27592,27594],{"className":27593},[476],[63,27595,27597,27628],{"className":27596},[411,412],[63,27598,27600,27625],{"className":27599},[416],[63,27601,27603,27614],{"className":27602,"style":25181},[420],[63,27604,27605,27608],{"style":25900},[63,27606],{"className":27607,"style":817},[428],[63,27609,27611],{"className":27610},[433,434,435,436],[63,27612,1057],{"className":27613},[399,472,436],[63,27615,27616,27619],{"style":16572},[63,27617],{"className":27618,"style":817},[428],[63,27620,27622],{"className":27621},[433,434,435,436],[63,27623,949],{"className":27624,"style":979},[399,472,436],[63,27626,507],{"className":27627},[506],[63,27629,27631],{"className":27630},[416],[63,27632,27634],{"className":27633,"style":17487},[420],[63,27635],{},[63,27637],{"className":27638,"style":1920},[776],[63,27640,5839],{"className":27641},[830],[63,27643],{"className":27644,"style":1920},[776],[63,27646,27648,27651,27654,27714,27717,27720],{"className":27647},[390],[63,27649],{"className":27650,"style":26651},[394],[63,27652,19142],{"className":27653,"style":19389},[399,472],[63,27655,27657,27660],{"className":27656},[399],[63,27658,25861],{"className":27659,"style":25885},[399,472],[63,27661,27663],{"className":27662},[476],[63,27664,27666,27706],{"className":27665},[411,412],[63,27667,27669,27703],{"className":27668},[416],[63,27670,27672,27692],{"className":27671,"style":25181},[420],[63,27673,27674,27677],{"style":25900},[63,27675],{"className":27676,"style":817},[428],[63,27678,27680],{"className":27679},[433,434,435,436],[63,27681,27683,27686,27689],{"className":27682},[399,436],[63,27684,1057],{"className":27685},[399,472,436],[63,27687,5839],{"className":27688},[830,436],[63,27690,363],{"className":27691},[399,436],[63,27693,27694,27697],{"style":16572},[63,27695],{"className":27696,"style":817},[428],[63,27698,27700],{"className":27699},[433,434,435,436],[63,27701,949],{"className":27702,"style":979},[399,472,436],[63,27704,507],{"className":27705},[506],[63,27707,27709],{"className":27708},[416],[63,27710,27712],{"className":27711,"style":26710},[420],[63,27713],{},[63,27715],{"className":27716,"style":1920},[776],[63,27718,5839],{"className":27719},[830],[63,27721],{"className":27722,"style":1920},[776],[63,27724,27726,27729,27758],{"className":27725},[390],[63,27727],{"className":27728,"style":26651},[394],[63,27730,27732,27735],{"className":27731},[399],[63,27733,19142],{"className":27734,"style":19389},[399,472],[63,27736,27738],{"className":27737},[476],[63,27739,27741],{"className":27740},[411],[63,27742,27744],{"className":27743},[416],[63,27745,27747],{"className":27746,"style":23695},[420],[63,27748,27749,27752],{"style":16572},[63,27750],{"className":27751,"style":817},[428],[63,27753,27755],{"className":27754},[433,434,435,436],[63,27756,2883],{"className":27757},[399,436],[63,27759,27761,27764],{"className":27760},[399],[63,27762,25861],{"className":27763,"style":25885},[399,472],[63,27765,27767],{"className":27766},[476],[63,27768,27770,27810],{"className":27769},[411,412],[63,27771,27773,27807],{"className":27772},[416],[63,27774,27776,27796],{"className":27775,"style":25181},[420],[63,27777,27778,27781],{"style":25900},[63,27779],{"className":27780,"style":817},[428],[63,27782,27784],{"className":27783},[433,434,435,436],[63,27785,27787,27790,27793],{"className":27786},[399,436],[63,27788,1057],{"className":27789},[399,472,436],[63,27791,5839],{"className":27792},[830,436],[63,27794,2883],{"className":27795},[399,436],[63,27797,27798,27801],{"style":16572},[63,27799],{"className":27800,"style":817},[428],[63,27802,27804],{"className":27803},[433,434,435,436],[63,27805,949],{"className":27806,"style":979},[399,472,436],[63,27808,507],{"className":27809},[506],[63,27811,27813],{"className":27812},[416],[63,27814,27816],{"className":27815,"style":26710},[420],[63,27817],{},[63,27819],{"className":27820},[776,21897],[63,27822,27824,27827],{"className":27823},[390],[63,27825],{"className":27826,"style":25141},[394],[63,27828,27830,27833],{"className":27829},[399],[63,27831,23523],{"className":27832},[399],[63,27834],{"className":27835,"style":25152},[399,25151],[30,27837,27839],{"id":27838},"generalized-advantage-estimation-gae","Generalized Advantage Estimation (GAE)",[14,27841,27842],{},"那么\"展开几步\"这件事到底该怎么选？GAE 给出的答案是：小孩子才做选择，全都要！",[14,27844,27845,27846,27905,27906,27936],{},"GAE 把不同展开步数的优势估计 ",[63,27847,27849,27867],{"className":27848},[342],[63,27850,27852],{"className":27851},[346],[348,27853,27854],{"xmlns":350},[352,27855,27856,27864],{},[355,27857,27858],{},[742,27859,27860,27862],{},[371,27861,1054],{},[371,27863,376],{},[378,27865,27866],{"encoding":380},"A^k",[63,27868,27870],{"className":27869,"ariaHidden":386},[385],[63,27871,27873,27876],{"className":27872},[390],[63,27874],{"className":27875,"style":791},[394],[63,27877,27879,27882],{"className":27878},[399],[63,27880,1054],{"className":27881},[399,472],[63,27883,27885],{"className":27884},[476],[63,27886,27888],{"className":27887},[411],[63,27889,27891],{"className":27890},[416],[63,27892,27894],{"className":27893,"style":791},[420],[63,27895,27896,27899],{"style":813},[63,27897],{"className":27898,"style":817},[428],[63,27900,27902],{"className":27901},[433,434,435,436],[63,27903,376],{"className":27904,"style":502},[399,472,436]," 用一个权重 ",[63,27907,27909,27924],{"className":27908},[342],[63,27910,27912],{"className":27911},[346],[348,27913,27914],{"xmlns":350},[352,27915,27916,27921],{},[355,27917,27918],{},[371,27919,27920],{},"λ",[378,27922,27923],{"encoding":380},"\\lambda",[63,27925,27927],{"className":27926,"ariaHidden":386},[385],[63,27928,27930,27933],{"className":27929},[390],[63,27931],{"className":27932,"style":684},[394],[63,27934,27920],{"className":27935},[399,472]," 做指数加权平均，再化简成一个简洁的求和形式：",[63,27938,27940],{"className":27939},[14495],[63,27941,27943,28533],{"className":27942},[342],[63,27944,27946],{"className":27945},[346],[348,27947,27948],{"xmlns":350,"display":9561},[352,27949,27950,28530],{},[14507,27951,27952,28129,28271,28381,28471],{"rowspacing":14509,"columnalign":14510,"columnspacing":14511},[14513,27953,27954,27989],{},[14516,27955,27956],{},[14519,27957,27958],{"scriptlevel":1802,"displaystyle":386},[355,27959,27960,27975,27977,27983,27985,27987],{},[1870,27961,27962,27964,27966],{},[371,27963,1054],{},[371,27965,14101],{},[355,27967,27968,27971,27973],{},[371,27969,27970],{},"G",[371,27972,1054],{},[371,27974,5688],{},[738,27976,1078],{"stretchy":1077},[368,27978,27979,27981],{},[371,27980,12890],{},[371,27982,1057],{},[738,27984,1083],{"separator":386},[371,27986,38],{},[738,27988,1092],{"stretchy":1077},[14516,27990,27991],{},[14519,27992,27993],{"scriptlevel":1802,"displaystyle":386},[355,27994,27995,27997,27999,28001,28003,28005,28007,28009,28011,28019,28021,28023,28025,28033,28035,28041,28049,28051,28053,28055,28057,28060,28062,28064,28066,28069,28071,28073,28087,28089,28091,28099,28101,28104,28112,28114,28117,28125,28127],{},[355,27996],{},[738,27998,204],{},[738,28000,1078],{"stretchy":1077},[361,28002,363],{},[738,28004,18511],{},[371,28006,27920],{},[738,28008,1092],{"stretchy":1077},[738,28010,1078],{"stretchy":1077},[1870,28012,28013,28015,28017],{},[371,28014,1054],{},[371,28016,14101],{},[361,28018,363],{},[738,28020,5839],{},[371,28022,27920],{},[738,28024,23434],{},[1870,28026,28027,28029,28031],{},[371,28028,1054],{},[371,28030,14101],{},[361,28032,2883],{},[738,28034,5839],{},[742,28036,28037,28039],{},[371,28038,27920],{},[361,28040,2883],{},[1870,28042,28043,28045,28047],{},[371,28044,1054],{},[371,28046,14101],{},[361,28048,4738],{},[738,28050,5839],{},[738,28052,23629],{},[3343,28054,7365],{},[738,28056,1092],{"stretchy":1077},[776,28058],{"width":28059},"1em",[776,28061],{"width":28059},[371,28063,27920],{},[738,28065,204],{},[361,28067,28068],{},"0.9",[738,28070,7202],{},[776,28072],{"width":28059},[1870,28074,28075,28077,28079],{},[371,28076,1054],{},[371,28078,14101],{},[355,28080,28081,28083,28085],{},[371,28082,27970],{},[371,28084,1054],{},[371,28086,5688],{},[738,28088,204],{},[361,28090,12548],{},[1870,28092,28093,28095,28097],{},[371,28094,1054],{},[371,28096,14101],{},[361,28098,363],{},[738,28100,5839],{},[361,28102,28103],{},"0.09",[1870,28105,28106,28108,28110],{},[371,28107,1054],{},[371,28109,14101],{},[361,28111,2883],{},[738,28113,5839],{},[361,28115,28116],{},"0.081",[1870,28118,28119,28121,28123],{},[371,28120,1054],{},[371,28122,14101],{},[361,28124,4738],{},[738,28126,5839],{},[738,28128,23629],{},[14513,28130,28131,28137],{},[14516,28132,28133],{},[14519,28134,28135],{"scriptlevel":1802,"displaystyle":386},[355,28136],{},[14516,28138,28139],{},[14519,28140,28141],{"scriptlevel":1802,"displaystyle":386},[355,28142,28143,28145,28147,28149,28151,28153,28155,28157,28159,28167,28169,28171,28173,28175,28183,28185,28187,28201,28203,28205,28211,28213,28221,28223,28225,28239,28241,28247,28261,28263,28265,28267,28269],{},[355,28144],{},[738,28146,204],{},[738,28148,1078],{"stretchy":1077},[361,28150,363],{},[738,28152,18511],{},[371,28154,27920],{},[738,28156,1092],{"stretchy":1077},[738,28158,1078],{"stretchy":1077},[1870,28160,28161,28163,28165],{},[371,28162,25861],{},[371,28164,1057],{},[371,28166,949],{},[738,28168,5839],{},[371,28170,27920],{},[738,28172,23434],{},[738,28174,1078],{"stretchy":1077},[1870,28176,28177,28179,28181],{},[371,28178,25861],{},[371,28180,1057],{},[371,28182,949],{},[738,28184,5839],{},[371,28186,19142],{},[1870,28188,28189,28191,28199],{},[371,28190,25861],{},[355,28192,28193,28195,28197],{},[371,28194,1057],{},[738,28196,5839],{},[361,28198,363],{},[371,28200,949],{},[738,28202,1092],{"stretchy":1077},[738,28204,5839],{},[742,28206,28207,28209],{},[371,28208,27920],{},[361,28210,2883],{},[738,28212,1078],{"stretchy":1077},[1870,28214,28215,28217,28219],{},[371,28216,25861],{},[371,28218,1057],{},[371,28220,949],{},[738,28222,5839],{},[371,28224,19142],{},[1870,28226,28227,28229,28237],{},[371,28228,25861],{},[355,28230,28231,28233,28235],{},[371,28232,1057],{},[738,28234,5839],{},[361,28236,363],{},[371,28238,949],{},[738,28240,5839],{},[742,28242,28243,28245],{},[371,28244,19142],{},[361,28246,2883],{},[1870,28248,28249,28251,28259],{},[371,28250,25861],{},[355,28252,28253,28255,28257],{},[371,28254,1057],{},[738,28256,5839],{},[361,28258,2883],{},[371,28260,949],{},[738,28262,1092],{"stretchy":1077},[738,28264,5839],{},[738,28266,23629],{},[3343,28268,7365],{},[738,28270,1092],{"stretchy":1077},[14513,28272,28273,28279],{},[14516,28274,28275],{},[14519,28276,28277],{"scriptlevel":1802,"displaystyle":386},[355,28278],{},[14516,28280,28281],{},[14519,28282,28283],{"scriptlevel":1802,"displaystyle":386},[355,28284,28285,28287,28289,28291,28293,28295,28297,28299,28301,28309,28311,28313,28315,28317,28319,28325,28327,28329,28331,28333,28335,28337,28351,28353,28355,28357,28359,28365,28367,28369,28371,28373,28375,28377,28379],{},[355,28286],{},[738,28288,204],{},[738,28290,1078],{"stretchy":1077},[361,28292,363],{},[738,28294,18511],{},[371,28296,27920],{},[738,28298,1092],{"stretchy":1077},[738,28300,1078],{"stretchy":1077},[1870,28302,28303,28305,28307],{},[371,28304,25861],{},[371,28306,1057],{},[371,28308,949],{},[738,28310,1078],{"stretchy":1077},[361,28312,363],{},[738,28314,5839],{},[371,28316,27920],{},[738,28318,5839],{},[742,28320,28321,28323],{},[371,28322,27920],{},[361,28324,2883],{},[738,28326,5839],{},[738,28328,23629],{},[3343,28330,7365],{},[738,28332,1092],{"stretchy":1077},[738,28334,5839],{},[371,28336,19142],{},[1870,28338,28339,28341,28349],{},[371,28340,25861],{},[355,28342,28343,28345,28347],{},[371,28344,1057],{},[738,28346,5839],{},[361,28348,363],{},[371,28350,949],{},[738,28352,23434],{},[738,28354,1078],{"stretchy":1077},[371,28356,27920],{},[738,28358,5839],{},[742,28360,28361,28363],{},[371,28362,27920],{},[361,28364,2883],{},[738,28366,5839],{},[738,28368,23629],{},[3343,28370,7365],{},[738,28372,1092],{"stretchy":1077},[738,28374,5839],{},[738,28376,23629],{},[3343,28378,7365],{},[738,28380,1092],{"stretchy":1077},[14513,28382,28383,28389],{},[14516,28384,28385],{},[14519,28386,28387],{"scriptlevel":1802,"displaystyle":386},[355,28388],{},[14516,28390,28391],{},[14519,28392,28393],{"scriptlevel":1802,"displaystyle":386},[355,28394,28395,28397,28399,28401,28403,28405,28407,28409],{},[355,28396],{},[738,28398,204],{},[738,28400,1078],{"stretchy":1077},[361,28402,363],{},[738,28404,18511],{},[371,28406,27920],{},[738,28408,1092],{"stretchy":1077},[355,28410,28411,28413,28421,28433,28435,28437,28451,28463,28465,28467,28469],{},[738,28412,1078],{"fence":386},[1870,28414,28415,28417,28419],{},[371,28416,25861],{},[371,28418,1057],{},[371,28420,949],{},[358,28422,28423,28425],{},[361,28424,363],{},[355,28426,28427,28429,28431],{},[361,28428,363],{},[738,28430,18511],{},[371,28432,27920],{},[738,28434,5839],{},[371,28436,19142],{},[1870,28438,28439,28441,28449],{},[371,28440,25861],{},[355,28442,28443,28445,28447],{},[371,28444,1057],{},[738,28446,5839],{},[361,28448,363],{},[371,28450,949],{},[358,28452,28453,28455],{},[371,28454,27920],{},[355,28456,28457,28459,28461],{},[361,28458,363],{},[738,28460,18511],{},[371,28462,27920],{},[738,28464,5839],{},[738,28466,23629],{},[3343,28468,7365],{},[738,28470,1092],{"fence":386},[14513,28472,28473,28479],{},[14516,28474,28475],{},[14519,28476,28477],{"scriptlevel":1802,"displaystyle":386},[355,28478],{},[14516,28480,28481],{},[14519,28482,28483],{"scriptlevel":1802,"displaystyle":386},[355,28484,28485,28487,28489,28504,28506,28508,28510,28516],{},[355,28486],{},[738,28488,204],{},[14828,28490,28491,28493,28501],{},[738,28492,1874],{},[355,28494,28495,28497,28499],{},[371,28496,7380],{},[738,28498,204],{},[361,28500,1802],{},[371,28502,28503],{"mathvariant":1053},"∞",[738,28505,1078],{"stretchy":1077},[371,28507,19142],{},[371,28509,27920],{},[742,28511,28512,28514],{},[738,28513,1092],{"stretchy":1077},[371,28515,7380],{},[1870,28517,28518,28520,28528],{},[371,28519,25861],{},[355,28521,28522,28524,28526],{},[371,28523,1057],{},[738,28525,5839],{},[371,28527,7380],{},[371,28529,949],{},[378,28531,28532],{"encoding":380},"\\begin{aligned}\nA_\\theta^{GAE}(s_t, a) &= (1 - \\lambda)(A_\\theta^1 + \\lambda * A_\\theta^2 + \\lambda^2 A_\\theta^3 + \\cdots) \\quad\\quad \\lambda = 0.9: \\quad A_\\theta^{GAE} = 0.1A_\\theta^1 + 0.09A_\\theta^2 + 0.081A_\\theta^3 + \\cdots \\\\\n&= (1 - \\lambda)(\\delta_t^V + \\lambda * (\\delta_t^V + \\gamma \\delta_{t+1}^V) + \\lambda^2 (\\delta_t^V + \\gamma \\delta_{t+1}^V + \\gamma^2 \\delta_{t+2}^V) + \\cdots) \\\\\n&= (1 - \\lambda)(\\delta_t^V (1 + \\lambda + \\lambda^2 + \\cdots) + \\gamma \\delta_{t+1}^V * (\\lambda + \\lambda^2 + \\cdots) + \\cdots) \\\\\n&= (1 - \\lambda)\\left(\\delta_t^V \\frac{1}{1 - \\lambda} + \\gamma \\delta_{t+1}^V \\frac{\\lambda}{1 - \\lambda} + \\cdots\\right) \\\\\n&= \\sum\\limits_{b=0}^\\infty (\\gamma\\lambda)^b \\delta_{t+b}^V\n\\end{aligned}",[63,28534,28536],{"className":28535,"ariaHidden":386},[385],[63,28537,28539,28543],{"className":28538},[390],[63,28540],{"className":28541,"style":28542},[394],"height:10.6075em;vertical-align:-5.0538em;",[63,28544,28546],{"className":28545},[399],[63,28547,28549,28735],{"className":28548},[14507],[63,28550,28552],{"className":28551},[15298],[63,28553,28555,28726],{"className":28554},[411,412],[63,28556,28558,28723],{"className":28557},[416],[63,28559,28562,28687,28696,28705,28714],{"className":28560,"style":28561},[420],"height:5.5538em;",[63,28563,28565,28569],{"style":28564},"top:-8.3138em;",[63,28566],{"className":28567,"style":28568},[428],"height:3.6514em;",[63,28570,28572,28632,28635,28675,28678,28681,28684],{"className":28571},[399],[63,28573,28575,28578],{"className":28574},[399],[63,28576,1054],{"className":28577},[399,472],[63,28579,28581],{"className":28580},[476],[63,28582,28584,28624],{"className":28583},[411,412],[63,28585,28587,28621],{"className":28586},[416],[63,28588,28590,28601],{"className":28589,"style":25181},[420],[63,28591,28592,28595],{"style":17457},[63,28593],{"className":28594,"style":817},[428],[63,28596,28598],{"className":28597},[433,434,435,436],[63,28599,14101],{"className":28600,"style":3276},[399,472,436],[63,28602,28603,28606],{"style":16572},[63,28604],{"className":28605,"style":817},[428],[63,28607,28609],{"className":28608},[433,434,435,436],[63,28610,28612,28615,28618],{"className":28611},[399,436],[63,28613,27970],{"className":28614},[399,472,436],[63,28616,1054],{"className":28617},[399,472,436],[63,28619,5688],{"className":28620,"style":6101},[399,472,436],[63,28622,507],{"className":28623},[506],[63,28625,28627],{"className":28626},[416],[63,28628,28630],{"className":28629,"style":17487},[420],[63,28631],{},[63,28633,1078],{"className":28634},[403],[63,28636,28638,28641],{"className":28637},[399],[63,28639,12890],{"className":28640},[399,472],[63,28642,28644],{"className":28643},[476],[63,28645,28647,28667],{"className":28646},[411,412],[63,28648,28650,28664],{"className":28649},[416],[63,28651,28653],{"className":28652,"style":12818},[420],[63,28654,28655,28658],{"style":1535},[63,28656],{"className":28657,"style":817},[428],[63,28659,28661],{"className":28660},[433,434,435,436],[63,28662,1057],{"className":28663},[399,472,436],[63,28665,507],{"className":28666},[506],[63,28668,28670],{"className":28669},[416],[63,28671,28673],{"className":28672,"style":1554},[420],[63,28674],{},[63,28676,1083],{"className":28677},[1173],[63,28679],{"className":28680,"style":1177},[776],[63,28682,38],{"className":28683},[399,472],[63,28685,1092],{"className":28686},[593],[63,28688,28690,28693],{"style":28689},"top:-6.7625em;",[63,28691],{"className":28692,"style":28568},[428],[63,28694],{"className":28695},[399],[63,28697,28699,28702],{"style":28698},"top:-5.2112em;",[63,28700],{"className":28701,"style":28568},[428],[63,28703],{"className":28704},[399],[63,28706,28708,28711],{"style":28707},"top:-3.1012em;",[63,28709],{"className":28710,"style":28568},[428],[63,28712],{"className":28713},[399],[63,28715,28717,28720],{"style":28716},"top:-0.1997em;",[63,28718],{"className":28719,"style":28568},[428],[63,28721],{"className":28722},[399],[63,28724,507],{"className":28725},[506],[63,28727,28729],{"className":28728},[416],[63,28730,28733],{"className":28731,"style":28732},[420],"height:5.0538em;",[63,28734],{},[63,28736,28738],{"className":28737},[15522],[63,28739,28741,30710],{"className":28740},[411,412],[63,28742,28744,30707],{"className":28743},[416],[63,28745,28747,29316,29844,30174,30521],{"className":28746,"style":28561},[420],[63,28748,28749,28752],{"style":28564},[63,28750],{"className":28751,"style":28568},[428],[63,28753,28755,28758,28761,28764,28767,28770,28773,28776,28779,28782,28785,28788,28791,28842,28845,28848,28851,28854,28857,28860,28863,28914,28917,28920,28923,28952,29003,29006,29009,29012,29015,29018,29021,29025,29028,29031,29034,29037,29040,29043,29046,29049,29052,29055,29115,29118,29121,29124,29127,29178,29181,29184,29187,29190,29241,29244,29247,29250,29253,29304,29307,29310,29313],{"className":28754},[399],[63,28756],{"className":28757},[399],[63,28759],{"className":28760,"style":777},[776],[63,28762,204],{"className":28763},[781],[63,28765],{"className":28766,"style":777},[776],[63,28768,1078],{"className":28769},[403],[63,28771,363],{"className":28772},[399],[63,28774],{"className":28775,"style":1920},[776],[63,28777,18511],{"className":28778},[830],[63,28780],{"className":28781,"style":1920},[776],[63,28783,27920],{"className":28784},[399,472],[63,28786,1092],{"className":28787},[593],[63,28789,1078],{"className":28790},[403],[63,28792,28794,28797],{"className":28793},[399],[63,28795,1054],{"className":28796},[399,472],[63,28798,28800],{"className":28799},[476],[63,28801,28803,28834],{"className":28802},[411,412],[63,28804,28806,28831],{"className":28805},[416],[63,28807,28809,28820],{"className":28808,"style":23695},[420],[63,28810,28811,28814],{"style":17457},[63,28812],{"className":28813,"style":817},[428],[63,28815,28817],{"className":28816},[433,434,435,436],[63,28818,14101],{"className":28819,"style":3276},[399,472,436],[63,28821,28822,28825],{"style":16572},[63,28823],{"className":28824,"style":817},[428],[63,28826,28828],{"className":28827},[433,434,435,436],[63,28829,363],{"className":28830},[399,436],[63,28832,507],{"className":28833},[506],[63,28835,28837],{"className":28836},[416],[63,28838,28840],{"className":28839,"style":17487},[420],[63,28841],{},[63,28843],{"className":28844,"style":1920},[776],[63,28846,5839],{"className":28847},[830],[63,28849],{"className":28850,"style":1920},[776],[63,28852,27920],{"className":28853},[399,472],[63,28855],{"className":28856,"style":1920},[776],[63,28858,23434],{"className":28859},[830],[63,28861],{"className":28862,"style":1920},[776],[63,28864,28866,28869],{"className":28865},[399],[63,28867,1054],{"className":28868},[399,472],[63,28870,28872],{"className":28871},[476],[63,28873,28875,28906],{"className":28874},[411,412],[63,28876,28878,28903],{"className":28877},[416],[63,28879,28881,28892],{"className":28880,"style":23695},[420],[63,28882,28883,28886],{"style":17457},[63,28884],{"className":28885,"style":817},[428],[63,28887,28889],{"className":28888},[433,434,435,436],[63,28890,14101],{"className":28891,"style":3276},[399,472,436],[63,28893,28894,28897],{"style":16572},[63,28895],{"className":28896,"style":817},[428],[63,28898,28900],{"className":28899},[433,434,435,436],[63,28901,2883],{"className":28902},[399,436],[63,28904,507],{"className":28905},[506],[63,28907,28909],{"className":28908},[416],[63,28910,28912],{"className":28911,"style":17487},[420],[63,28913],{},[63,28915],{"className":28916,"style":1920},[776],[63,28918,5839],{"className":28919},[830],[63,28921],{"className":28922,"style":1920},[776],[63,28924,28926,28929],{"className":28925},[399],[63,28927,27920],{"className":28928},[399,472],[63,28930,28932],{"className":28931},[476],[63,28933,28935],{"className":28934},[411],[63,28936,28938],{"className":28937},[416],[63,28939,28941],{"className":28940,"style":23695},[420],[63,28942,28943,28946],{"style":16572},[63,28944],{"className":28945,"style":817},[428],[63,28947,28949],{"className":28948},[433,434,435,436],[63,28950,2883],{"className":28951},[399,436],[63,28953,28955,28958],{"className":28954},[399],[63,28956,1054],{"className":28957},[399,472],[63,28959,28961],{"className":28960},[476],[63,28962,28964,28995],{"className":28963},[411,412],[63,28965,28967,28992],{"className":28966},[416],[63,28968,28970,28981],{"className":28969,"style":23695},[420],[63,28971,28972,28975],{"style":17457},[63,28973],{"className":28974,"style":817},[428],[63,28976,28978],{"className":28977},[433,434,435,436],[63,28979,14101],{"className":28980,"style":3276},[399,472,436],[63,28982,28983,28986],{"style":16572},[63,28984],{"className":28985,"style":817},[428],[63,28987,28989],{"className":28988},[433,434,435,436],[63,28990,4738],{"className":28991},[399,436],[63,28993,507],{"className":28994},[506],[63,28996,28998],{"className":28997},[416],[63,28999,29001],{"className":29000,"style":17487},[420],[63,29002],{},[63,29004],{"className":29005,"style":1920},[776],[63,29007,5839],{"className":29008},[830],[63,29010],{"className":29011,"style":1920},[776],[63,29013,23629],{"className":29014},[1222],[63,29016],{"className":29017,"style":1177},[776],[63,29019,1092],{"className":29020},[593],[63,29022],{"className":29023,"style":29024},[776],"margin-right:1em;",[63,29026],{"className":29027,"style":29024},[776],[63,29029,27920],{"className":29030},[399,472],[63,29032],{"className":29033,"style":777},[776],[63,29035,204],{"className":29036},[781],[63,29038],{"className":29039,"style":777},[776],[63,29041,28068],{"className":29042},[399],[63,29044],{"className":29045,"style":777},[776],[63,29047,7202],{"className":29048},[781],[63,29050],{"className":29051,"style":29024},[776],[63,29053],{"className":29054,"style":777},[776],[63,29056,29058,29061],{"className":29057},[399],[63,29059,1054],{"className":29060},[399,472],[63,29062,29064],{"className":29063},[476],[63,29065,29067,29107],{"className":29066},[411,412],[63,29068,29070,29104],{"className":29069},[416],[63,29071,29073,29084],{"className":29072,"style":25181},[420],[63,29074,29075,29078],{"style":17457},[63,29076],{"className":29077,"style":817},[428],[63,29079,29081],{"className":29080},[433,434,435,436],[63,29082,14101],{"className":29083,"style":3276},[399,472,436],[63,29085,29086,29089],{"style":16572},[63,29087],{"className":29088,"style":817},[428],[63,29090,29092],{"className":29091},[433,434,435,436],[63,29093,29095,29098,29101],{"className":29094},[399,436],[63,29096,27970],{"className":29097},[399,472,436],[63,29099,1054],{"className":29100},[399,472,436],[63,29102,5688],{"className":29103,"style":6101},[399,472,436],[63,29105,507],{"className":29106},[506],[63,29108,29110],{"className":29109},[416],[63,29111,29113],{"className":29112,"style":17487},[420],[63,29114],{},[63,29116],{"className":29117,"style":777},[776],[63,29119,204],{"className":29120},[781],[63,29122],{"className":29123,"style":777},[776],[63,29125,12548],{"className":29126},[399],[63,29128,29130,29133],{"className":29129},[399],[63,29131,1054],{"className":29132},[399,472],[63,29134,29136],{"className":29135},[476],[63,29137,29139,29170],{"className":29138},[411,412],[63,29140,29142,29167],{"className":29141},[416],[63,29143,29145,29156],{"className":29144,"style":23695},[420],[63,29146,29147,29150],{"style":17457},[63,29148],{"className":29149,"style":817},[428],[63,29151,29153],{"className":29152},[433,434,435,436],[63,29154,14101],{"className":29155,"style":3276},[399,472,436],[63,29157,29158,29161],{"style":16572},[63,29159],{"className":29160,"style":817},[428],[63,29162,29164],{"className":29163},[433,434,435,436],[63,29165,363],{"className":29166},[399,436],[63,29168,507],{"className":29169},[506],[63,29171,29173],{"className":29172},[416],[63,29174,29176],{"className":29175,"style":17487},[420],[63,29177],{},[63,29179],{"className":29180,"style":1920},[776],[63,29182,5839],{"className":29183},[830],[63,29185],{"className":29186,"style":1920},[776],[63,29188,28103],{"className":29189},[399],[63,29191,29193,29196],{"className":29192},[399],[63,29194,1054],{"className":29195},[399,472],[63,29197,29199],{"className":29198},[476],[63,29200,29202,29233],{"className":29201},[411,412],[63,29203,29205,29230],{"className":29204},[416],[63,29206,29208,29219],{"className":29207,"style":23695},[420],[63,29209,29210,29213],{"style":17457},[63,29211],{"className":29212,"style":817},[428],[63,29214,29216],{"className":29215},[433,434,435,436],[63,29217,14101],{"className":29218,"style":3276},[399,472,436],[63,29220,29221,29224],{"style":16572},[63,29222],{"className":29223,"style":817},[428],[63,29225,29227],{"className":29226},[433,434,435,436],[63,29228,2883],{"className":29229},[399,436],[63,29231,507],{"className":29232},[506],[63,29234,29236],{"className":29235},[416],[63,29237,29239],{"className":29238,"style":17487},[420],[63,29240],{},[63,29242],{"className":29243,"style":1920},[776],[63,29245,5839],{"className":29246},[830],[63,29248],{"className":29249,"style":1920},[776],[63,29251,28116],{"className":29252},[399],[63,29254,29256,29259],{"className":29255},[399],[63,29257,1054],{"className":29258},[399,472],[63,29260,29262],{"className":29261},[476],[63,29263,29265,29296],{"className":29264},[411,412],[63,29266,29268,29293],{"className":29267},[416],[63,29269,29271,29282],{"className":29270,"style":23695},[420],[63,29272,29273,29276],{"style":17457},[63,29274],{"className":29275,"style":817},[428],[63,29277,29279],{"className":29278},[433,434,435,436],[63,29280,14101],{"className":29281,"style":3276},[399,472,436],[63,29283,29284,29287],{"style":16572},[63,29285],{"className":29286,"style":817},[428],[63,29288,29290],{"className":29289},[433,434,435,436],[63,29291,4738],{"className":29292},[399,436],[63,29294,507],{"className":29295},[506],[63,29297,29299],{"className":29298},[416],[63,29300,29302],{"className":29301,"style":17487},[420],[63,29303],{},[63,29305],{"className":29306,"style":1920},[776],[63,29308,5839],{"className":29309},[830],[63,29311],{"className":29312,"style":1920},[776],[63,29314,23629],{"className":29315},[1222],[63,29317,29318,29321],{"style":28689},[63,29319],{"className":29320,"style":28568},[428],[63,29322,29324,29327,29330,29333,29336,29339,29342,29345,29348,29351,29354,29357,29360,29411,29414,29417,29420,29423,29426,29429,29432,29435,29486,29489,29492,29495,29498,29558,29561,29564,29567,29570,29599,29602,29653,29656,29659,29662,29665,29725,29728,29731,29734,29763,29823,29826,29829,29832,29835,29838,29841],{"className":29323},[399],[63,29325],{"className":29326},[399],[63,29328],{"className":29329,"style":777},[776],[63,29331,204],{"className":29332},[781],[63,29334],{"className":29335,"style":777},[776],[63,29337,1078],{"className":29338},[403],[63,29340,363],{"className":29341},[399],[63,29343],{"className":29344,"style":1920},[776],[63,29346,18511],{"className":29347},[830],[63,29349],{"className":29350,"style":1920},[776],[63,29352,27920],{"className":29353},[399,472],[63,29355,1092],{"className":29356},[593],[63,29358,1078],{"className":29359},[403],[63,29361,29363,29366],{"className":29362},[399],[63,29364,25861],{"className":29365,"style":25885},[399,472],[63,29367,29369],{"className":29368},[476],[63,29370,29372,29403],{"className":29371},[411,412],[63,29373,29375,29400],{"className":29374},[416],[63,29376,29378,29389],{"className":29377,"style":25181},[420],[63,29379,29380,29383],{"style":25900},[63,29381],{"className":29382,"style":817},[428],[63,29384,29386],{"className":29385},[433,434,435,436],[63,29387,1057],{"className":29388},[399,472,436],[63,29390,29391,29394],{"style":16572},[63,29392],{"className":29393,"style":817},[428],[63,29395,29397],{"className":29396},[433,434,435,436],[63,29398,949],{"className":29399,"style":979},[399,472,436],[63,29401,507],{"className":29402},[506],[63,29404,29406],{"className":29405},[416],[63,29407,29409],{"className":29408,"style":17487},[420],[63,29410],{},[63,29412],{"className":29413,"style":1920},[776],[63,29415,5839],{"className":29416},[830],[63,29418],{"className":29419,"style":1920},[776],[63,29421,27920],{"className":29422},[399,472],[63,29424],{"className":29425,"style":1920},[776],[63,29427,23434],{"className":29428},[830],[63,29430],{"className":29431,"style":1920},[776],[63,29433,1078],{"className":29434},[403],[63,29436,29438,29441],{"className":29437},[399],[63,29439,25861],{"className":29440,"style":25885},[399,472],[63,29442,29444],{"className":29443},[476],[63,29445,29447,29478],{"className":29446},[411,412],[63,29448,29450,29475],{"className":29449},[416],[63,29451,29453,29464],{"className":29452,"style":25181},[420],[63,29454,29455,29458],{"style":25900},[63,29456],{"className":29457,"style":817},[428],[63,29459,29461],{"className":29460},[433,434,435,436],[63,29462,1057],{"className":29463},[399,472,436],[63,29465,29466,29469],{"style":16572},[63,29467],{"className":29468,"style":817},[428],[63,29470,29472],{"className":29471},[433,434,435,436],[63,29473,949],{"className":29474,"style":979},[399,472,436],[63,29476,507],{"className":29477},[506],[63,29479,29481],{"className":29480},[416],[63,29482,29484],{"className":29483,"style":17487},[420],[63,29485],{},[63,29487],{"className":29488,"style":1920},[776],[63,29490,5839],{"className":29491},[830],[63,29493],{"className":29494,"style":1920},[776],[63,29496,19142],{"className":29497,"style":19389},[399,472],[63,29499,29501,29504],{"className":29500},[399],[63,29502,25861],{"className":29503,"style":25885},[399,472],[63,29505,29507],{"className":29506},[476],[63,29508,29510,29550],{"className":29509},[411,412],[63,29511,29513,29547],{"className":29512},[416],[63,29514,29516,29536],{"className":29515,"style":25181},[420],[63,29517,29518,29521],{"style":25900},[63,29519],{"className":29520,"style":817},[428],[63,29522,29524],{"className":29523},[433,434,435,436],[63,29525,29527,29530,29533],{"className":29526},[399,436],[63,29528,1057],{"className":29529},[399,472,436],[63,29531,5839],{"className":29532},[830,436],[63,29534,363],{"className":29535},[399,436],[63,29537,29538,29541],{"style":16572},[63,29539],{"className":29540,"style":817},[428],[63,29542,29544],{"className":29543},[433,434,435,436],[63,29545,949],{"className":29546,"style":979},[399,472,436],[63,29548,507],{"className":29549},[506],[63,29551,29553],{"className":29552},[416],[63,29554,29556],{"className":29555,"style":26710},[420],[63,29557],{},[63,29559,1092],{"className":29560},[593],[63,29562],{"className":29563,"style":1920},[776],[63,29565,5839],{"className":29566},[830],[63,29568],{"className":29569,"style":1920},[776],[63,29571,29573,29576],{"className":29572},[399],[63,29574,27920],{"className":29575},[399,472],[63,29577,29579],{"className":29578},[476],[63,29580,29582],{"className":29581},[411],[63,29583,29585],{"className":29584},[416],[63,29586,29588],{"className":29587,"style":23695},[420],[63,29589,29590,29593],{"style":16572},[63,29591],{"className":29592,"style":817},[428],[63,29594,29596],{"className":29595},[433,434,435,436],[63,29597,2883],{"className":29598},[399,436],[63,29600,1078],{"className":29601},[403],[63,29603,29605,29608],{"className":29604},[399],[63,29606,25861],{"className":29607,"style":25885},[399,472],[63,29609,29611],{"className":29610},[476],[63,29612,29614,29645],{"className":29613},[411,412],[63,29615,29617,29642],{"className":29616},[416],[63,29618,29620,29631],{"className":29619,"style":25181},[420],[63,29621,29622,29625],{"style":25900},[63,29623],{"className":29624,"style":817},[428],[63,29626,29628],{"className":29627},[433,434,435,436],[63,29629,1057],{"className":29630},[399,472,436],[63,29632,29633,29636],{"style":16572},[63,29634],{"className":29635,"style":817},[428],[63,29637,29639],{"className":29638},[433,434,435,436],[63,29640,949],{"className":29641,"style":979},[399,472,436],[63,29643,507],{"className":29644},[506],[63,29646,29648],{"className":29647},[416],[63,29649,29651],{"className":29650,"style":17487},[420],[63,29652],{},[63,29654],{"className":29655,"style":1920},[776],[63,29657,5839],{"className":29658},[830],[63,29660],{"className":29661,"style":1920},[776],[63,29663,19142],{"className":29664,"style":19389},[399,472],[63,29666,29668,29671],{"className":29667},[399],[63,29669,25861],{"className":29670,"style":25885},[399,472],[63,29672,29674],{"className":29673},[476],[63,29675,29677,29717],{"className":29676},[411,412],[63,29678,29680,29714],{"className":29679},[416],[63,29681,29683,29703],{"className":29682,"style":25181},[420],[63,29684,29685,29688],{"style":25900},[63,29686],{"className":29687,"style":817},[428],[63,29689,29691],{"className":29690},[433,434,435,436],[63,29692,29694,29697,29700],{"className":29693},[399,436],[63,29695,1057],{"className":29696},[399,472,436],[63,29698,5839],{"className":29699},[830,436],[63,29701,363],{"className":29702},[399,436],[63,29704,29705,29708],{"style":16572},[63,29706],{"className":29707,"style":817},[428],[63,29709,29711],{"className":29710},[433,434,435,436],[63,29712,949],{"className":29713,"style":979},[399,472,436],[63,29715,507],{"className":29716},[506],[63,29718,29720],{"className":29719},[416],[63,29721,29723],{"className":29722,"style":26710},[420],[63,29724],{},[63,29726],{"className":29727,"style":1920},[776],[63,29729,5839],{"className":29730},[830],[63,29732],{"className":29733,"style":1920},[776],[63,29735,29737,29740],{"className":29736},[399],[63,29738,19142],{"className":29739,"style":19389},[399,472],[63,29741,29743],{"className":29742},[476],[63,29744,29746],{"className":29745},[411],[63,29747,29749],{"className":29748},[416],[63,29750,29752],{"className":29751,"style":23695},[420],[63,29753,29754,29757],{"style":16572},[63,29755],{"className":29756,"style":817},[428],[63,29758,29760],{"className":29759},[433,434,435,436],[63,29761,2883],{"className":29762},[399,436],[63,29764,29766,29769],{"className":29765},[399],[63,29767,25861],{"className":29768,"style":25885},[399,472],[63,29770,29772],{"className":29771},[476],[63,29773,29775,29815],{"className":29774},[411,412],[63,29776,29778,29812],{"className":29777},[416],[63,29779,29781,29801],{"className":29780,"style":25181},[420],[63,29782,29783,29786],{"style":25900},[63,29784],{"className":29785,"style":817},[428],[63,29787,29789],{"className":29788},[433,434,435,436],[63,29790,29792,29795,29798],{"className":29791},[399,436],[63,29793,1057],{"className":29794},[399,472,436],[63,29796,5839],{"className":29797},[830,436],[63,29799,2883],{"className":29800},[399,436],[63,29802,29803,29806],{"style":16572},[63,29804],{"className":29805,"style":817},[428],[63,29807,29809],{"className":29808},[433,434,435,436],[63,29810,949],{"className":29811,"style":979},[399,472,436],[63,29813,507],{"className":29814},[506],[63,29816,29818],{"className":29817},[416],[63,29819,29821],{"className":29820,"style":26710},[420],[63,29822],{},[63,29824,1092],{"className":29825},[593],[63,29827],{"className":29828,"style":1920},[776],[63,29830,5839],{"className":29831},[830],[63,29833],{"className":29834,"style":1920},[776],[63,29836,23629],{"className":29837},[1222],[63,29839],{"className":29840,"style":1177},[776],[63,29842,1092],{"className":29843},[593],[63,29845,29846,29849],{"style":28698},[63,29847],{"className":29848,"style":28568},[428],[63,29850,29852,29855,29858,29861,29864,29867,29870,29873,29876,29879,29882,29885,29888,29939,29942,29945,29948,29951,29954,29957,29960,29963,29966,29995,29998,30001,30004,30007,30010,30013,30016,30019,30022,30025,30085,30088,30091,30094,30097,30100,30103,30106,30109,30138,30141,30144,30147,30150,30153,30156,30159,30162,30165,30168,30171],{"className":29851},[399],[63,29853],{"className":29854},[399],[63,29856],{"className":29857,"style":777},[776],[63,29859,204],{"className":29860},[781],[63,29862],{"className":29863,"style":777},[776],[63,29865,1078],{"className":29866},[403],[63,29868,363],{"className":29869},[399],[63,29871],{"className":29872,"style":1920},[776],[63,29874,18511],{"className":29875},[830],[63,29877],{"className":29878,"style":1920},[776],[63,29880,27920],{"className":29881},[399,472],[63,29883,1092],{"className":29884},[593],[63,29886,1078],{"className":29887},[403],[63,29889,29891,29894],{"className":29890},[399],[63,29892,25861],{"className":29893,"style":25885},[399,472],[63,29895,29897],{"className":29896},[476],[63,29898,29900,29931],{"className":29899},[411,412],[63,29901,29903,29928],{"className":29902},[416],[63,29904,29906,29917],{"className":29905,"style":25181},[420],[63,29907,29908,29911],{"style":25900},[63,29909],{"className":29910,"style":817},[428],[63,29912,29914],{"className":29913},[433,434,435,436],[63,29915,1057],{"className":29916},[399,472,436],[63,29918,29919,29922],{"style":16572},[63,29920],{"className":29921,"style":817},[428],[63,29923,29925],{"className":29924},[433,434,435,436],[63,29926,949],{"className":29927,"style":979},[399,472,436],[63,29929,507],{"className":29930},[506],[63,29932,29934],{"className":29933},[416],[63,29935,29937],{"className":29936,"style":17487},[420],[63,29938],{},[63,29940,1078],{"className":29941},[403],[63,29943,363],{"className":29944},[399],[63,29946],{"className":29947,"style":1920},[776],[63,29949,5839],{"className":29950},[830],[63,29952],{"className":29953,"style":1920},[776],[63,29955,27920],{"className":29956},[399,472],[63,29958],{"className":29959,"style":1920},[776],[63,29961,5839],{"className":29962},[830],[63,29964],{"className":29965,"style":1920},[776],[63,29967,29969,29972],{"className":29968},[399],[63,29970,27920],{"className":29971},[399,472],[63,29973,29975],{"className":29974},[476],[63,29976,29978],{"className":29977},[411],[63,29979,29981],{"className":29980},[416],[63,29982,29984],{"className":29983,"style":23695},[420],[63,29985,29986,29989],{"style":16572},[63,29987],{"className":29988,"style":817},[428],[63,29990,29992],{"className":29991},[433,434,435,436],[63,29993,2883],{"className":29994},[399,436],[63,29996],{"className":29997,"style":1920},[776],[63,29999,5839],{"className":30000},[830],[63,30002],{"className":30003,"style":1920},[776],[63,30005,23629],{"className":30006},[1222],[63,30008],{"className":30009,"style":1177},[776],[63,30011,1092],{"className":30012},[593],[63,30014],{"className":30015,"style":1920},[776],[63,30017,5839],{"className":30018},[830],[63,30020],{"className":30021,"style":1920},[776],[63,30023,19142],{"className":30024,"style":19389},[399,472],[63,30026,30028,30031],{"className":30027},[399],[63,30029,25861],{"className":30030,"style":25885},[399,472],[63,30032,30034],{"className":30033},[476],[63,30035,30037,30077],{"className":30036},[411,412],[63,30038,30040,30074],{"className":30039},[416],[63,30041,30043,30063],{"className":30042,"style":25181},[420],[63,30044,30045,30048],{"style":25900},[63,30046],{"className":30047,"style":817},[428],[63,30049,30051],{"className":30050},[433,434,435,436],[63,30052,30054,30057,30060],{"className":30053},[399,436],[63,30055,1057],{"className":30056},[399,472,436],[63,30058,5839],{"className":30059},[830,436],[63,30061,363],{"className":30062},[399,436],[63,30064,30065,30068],{"style":16572},[63,30066],{"className":30067,"style":817},[428],[63,30069,30071],{"className":30070},[433,434,435,436],[63,30072,949],{"className":30073,"style":979},[399,472,436],[63,30075,507],{"className":30076},[506],[63,30078,30080],{"className":30079},[416],[63,30081,30083],{"className":30082,"style":26710},[420],[63,30084],{},[63,30086],{"className":30087,"style":1920},[776],[63,30089,23434],{"className":30090},[830],[63,30092],{"className":30093,"style":1920},[776],[63,30095,1078],{"className":30096},[403],[63,30098,27920],{"className":30099},[399,472],[63,30101],{"className":30102,"style":1920},[776],[63,30104,5839],{"className":30105},[830],[63,30107],{"className":30108,"style":1920},[776],[63,30110,30112,30115],{"className":30111},[399],[63,30113,27920],{"className":30114},[399,472],[63,30116,30118],{"className":30117},[476],[63,30119,30121],{"className":30120},[411],[63,30122,30124],{"className":30123},[416],[63,30125,30127],{"className":30126,"style":23695},[420],[63,30128,30129,30132],{"style":16572},[63,30130],{"className":30131,"style":817},[428],[63,30133,30135],{"className":30134},[433,434,435,436],[63,30136,2883],{"className":30137},[399,436],[63,30139],{"className":30140,"style":1920},[776],[63,30142,5839],{"className":30143},[830],[63,30145],{"className":30146,"style":1920},[776],[63,30148,23629],{"className":30149},[1222],[63,30151],{"className":30152,"style":1177},[776],[63,30154,1092],{"className":30155},[593],[63,30157],{"className":30158,"style":1920},[776],[63,30160,5839],{"className":30161},[830],[63,30163],{"className":30164,"style":1920},[776],[63,30166,23629],{"className":30167},[1222],[63,30169],{"className":30170,"style":1177},[776],[63,30172,1092],{"className":30173},[593],[63,30175,30176,30179],{"style":28707},[63,30177],{"className":30178,"style":28568},[428],[63,30180,30182,30185,30188,30191,30194,30197,30200,30203,30206,30209,30212,30215,30218],{"className":30181},[399],[63,30183],{"className":30184},[399],[63,30186],{"className":30187,"style":777},[776],[63,30189,204],{"className":30190},[781],[63,30192],{"className":30193,"style":777},[776],[63,30195,1078],{"className":30196},[403],[63,30198,363],{"className":30199},[399],[63,30201],{"className":30202,"style":1920},[776],[63,30204,18511],{"className":30205},[830],[63,30207],{"className":30208,"style":1920},[776],[63,30210,27920],{"className":30211},[399,472],[63,30213,1092],{"className":30214},[593],[63,30216],{"className":30217,"style":1177},[776],[63,30219,30221,30227,30278,30353,30356,30359,30362,30365,30425,30500,30503,30506,30509,30512,30515],{"className":30220},[1222],[63,30222,30224],{"className":30223,"style":1227},[403,1226],[63,30225,1078],{"className":30226},[1231,435],[63,30228,30230,30233],{"className":30229},[399],[63,30231,25861],{"className":30232,"style":25885},[399,472],[63,30234,30236],{"className":30235},[476],[63,30237,30239,30270],{"className":30238},[411,412],[63,30240,30242,30267],{"className":30241},[416],[63,30243,30245,30256],{"className":30244,"style":25181},[420],[63,30246,30247,30250],{"style":25900},[63,30248],{"className":30249,"style":817},[428],[63,30251,30253],{"className":30252},[433,434,435,436],[63,30254,1057],{"className":30255},[399,472,436],[63,30257,30258,30261],{"style":16572},[63,30259],{"className":30260,"style":817},[428],[63,30262,30264],{"className":30263},[433,434,435,436],[63,30265,949],{"className":30266,"style":979},[399,472,436],[63,30268,507],{"className":30269},[506],[63,30271,30273],{"className":30272},[416],[63,30274,30276],{"className":30275,"style":17487},[420],[63,30277],{},[63,30279,30281,30284,30350],{"className":30280},[399],[63,30282],{"className":30283},[403,404],[63,30285,30287],{"className":30286},[358],[63,30288,30290,30341],{"className":30289},[411,412],[63,30291,30293,30338],{"className":30292},[416],[63,30294,30296,30319,30327],{"className":30295,"style":16424},[420],[63,30297,30298,30301],{"style":15964},[63,30299],{"className":30300,"style":429},[428],[63,30302,30304,30307,30310,30313,30316],{"className":30303},[399],[63,30305,363],{"className":30306},[399],[63,30308],{"className":30309,"style":1920},[776],[63,30311,18511],{"className":30312},[830],[63,30314],{"className":30315,"style":1920},[776],[63,30317,27920],{"className":30318},[399,472],[63,30320,30321,30324],{"style":554},[63,30322],{"className":30323,"style":429},[428],[63,30325],{"className":30326,"style":562},[561],[63,30328,30329,30332],{"style":16030},[63,30330],{"className":30331,"style":429},[428],[63,30333,30335],{"className":30334},[399],[63,30336,363],{"className":30337},[399],[63,30339,507],{"className":30340},[506],[63,30342,30344],{"className":30343},[416],[63,30345,30348],{"className":30346,"style":30347},[420],"height:0.7693em;",[63,30349],{},[63,30351],{"className":30352},[593,404],[63,30354],{"className":30355,"style":1920},[776],[63,30357,5839],{"className":30358},[830],[63,30360],{"className":30361,"style":1920},[776],[63,30363,19142],{"className":30364,"style":19389},[399,472],[63,30366,30368,30371],{"className":30367},[399],[63,30369,25861],{"className":30370,"style":25885},[399,472],[63,30372,30374],{"className":30373},[476],[63,30375,30377,30417],{"className":30376},[411,412],[63,30378,30380,30414],{"className":30379},[416],[63,30381,30383,30403],{"className":30382,"style":25181},[420],[63,30384,30385,30388],{"style":25900},[63,30386],{"className":30387,"style":817},[428],[63,30389,30391],{"className":30390},[433,434,435,436],[63,30392,30394,30397,30400],{"className":30393},[399,436],[63,30395,1057],{"className":30396},[399,472,436],[63,30398,5839],{"className":30399},[830,436],[63,30401,363],{"className":30402},[399,436],[63,30404,30405,30408],{"style":16572},[63,30406],{"className":30407,"style":817},[428],[63,30409,30411],{"className":30410},[433,434,435,436],[63,30412,949],{"className":30413,"style":979},[399,472,436],[63,30415,507],{"className":30416},[506],[63,30418,30420],{"className":30419},[416],[63,30421,30423],{"className":30422,"style":26710},[420],[63,30424],{},[63,30426,30428,30431,30497],{"className":30427},[399],[63,30429],{"className":30430},[403,404],[63,30432,30434],{"className":30433},[358],[63,30435,30437,30489],{"className":30436},[411,412],[63,30438,30440,30486],{"className":30439},[416],[63,30441,30444,30467,30475],{"className":30442,"style":30443},[420],"height:1.3714em;",[63,30445,30446,30449],{"style":15964},[63,30447],{"className":30448,"style":429},[428],[63,30450,30452,30455,30458,30461,30464],{"className":30451},[399],[63,30453,363],{"className":30454},[399],[63,30456],{"className":30457,"style":1920},[776],[63,30459,18511],{"className":30460},[830],[63,30462],{"className":30463,"style":1920},[776],[63,30465,27920],{"className":30466},[399,472],[63,30468,30469,30472],{"style":554},[63,30470],{"className":30471,"style":429},[428],[63,30473],{"className":30474,"style":562},[561],[63,30476,30477,30480],{"style":16030},[63,30478],{"className":30479,"style":429},[428],[63,30481,30483],{"className":30482},[399],[63,30484,27920],{"className":30485},[399,472],[63,30487,507],{"className":30488},[506],[63,30490,30492],{"className":30491},[416],[63,30493,30495],{"className":30494,"style":30347},[420],[63,30496],{},[63,30498],{"className":30499},[593,404],[63,30501],{"className":30502,"style":1920},[776],[63,30504,5839],{"className":30505},[830],[63,30507],{"className":30508,"style":1920},[776],[63,30510,23629],{"className":30511},[1222],[63,30513],{"className":30514,"style":1177},[776],[63,30516,30518],{"className":30517,"style":1227},[593,1226],[63,30519,1092],{"className":30520},[1231,435],[63,30522,30523,30526],{"style":28716},[63,30524],{"className":30525,"style":28568},[428],[63,30527,30529,30532,30535,30538,30541,30608,30611,30614,30617,30647],{"className":30528},[399],[63,30530],{"className":30531},[399],[63,30533],{"className":30534,"style":777},[776],[63,30536,204],{"className":30537},[781],[63,30539],{"className":30540,"style":777},[776],[63,30542,30544],{"className":30543},[1212,14383],[63,30545,30547,30599],{"className":30546},[411,412],[63,30548,30550,30596],{"className":30549},[416],[63,30551,30554,30575,30585],{"className":30552,"style":30553},[420],"height:1.6514em;",[63,30555,30557,30560],{"style":30556},"top:-1.8479em;margin-left:0em;",[63,30558],{"className":30559,"style":15577},[428],[63,30561,30563],{"className":30562},[433,434,435,436],[63,30564,30566,30569,30572],{"className":30565},[399,436],[63,30567,7380],{"className":30568},[399,472,436],[63,30570,204],{"className":30571},[781,436],[63,30573,1802],{"className":30574},[399,436],[63,30576,30577,30580],{"style":15586},[63,30578],{"className":30579,"style":15577},[428],[63,30581,30582],{},[63,30583,1874],{"className":30584},[1212,1958,15595],[63,30586,30587,30590],{"style":16520},[63,30588],{"className":30589,"style":15577},[428],[63,30591,30593],{"className":30592},[433,434,435,436],[63,30594,28503],{"className":30595},[399,436],[63,30597,507],{"className":30598},[506],[63,30600,30602],{"className":30601},[416],[63,30603,30606],{"className":30604,"style":30605},[420],"height:1.3021em;",[63,30607],{},[63,30609,1078],{"className":30610},[403],[63,30612,19142],{"className":30613,"style":19389},[399,472],[63,30615,27920],{"className":30616},[399,472],[63,30618,30620,30623],{"className":30619},[593],[63,30621,1092],{"className":30622},[593],[63,30624,30626],{"className":30625},[476],[63,30627,30629],{"className":30628},[411],[63,30630,30632],{"className":30631},[416],[63,30633,30636],{"className":30634,"style":30635},[420],"height:0.8991em;",[63,30637,30638,30641],{"style":16572},[63,30639],{"className":30640,"style":817},[428],[63,30642,30644],{"className":30643},[433,434,435,436],[63,30645,7380],{"className":30646},[399,472,436],[63,30648,30650,30653],{"className":30649},[399],[63,30651,25861],{"className":30652,"style":25885},[399,472],[63,30654,30656],{"className":30655},[476],[63,30657,30659,30699],{"className":30658},[411,412],[63,30660,30662,30696],{"className":30661},[416],[63,30663,30665,30685],{"className":30664,"style":25181},[420],[63,30666,30667,30670],{"style":25900},[63,30668],{"className":30669,"style":817},[428],[63,30671,30673],{"className":30672},[433,434,435,436],[63,30674,30676,30679,30682],{"className":30675},[399,436],[63,30677,1057],{"className":30678},[399,472,436],[63,30680,5839],{"className":30681},[830,436],[63,30683,7380],{"className":30684},[399,472,436],[63,30686,30687,30690],{"style":16572},[63,30688],{"className":30689,"style":817},[428],[63,30691,30693],{"className":30692},[433,434,435,436],[63,30694,949],{"className":30695,"style":979},[399,472,436],[63,30697,507],{"className":30698},[506],[63,30700,30702],{"className":30701},[416],[63,30703,30705],{"className":30704,"style":26710},[420],[63,30706],{},[63,30708,507],{"className":30709},[506],[63,30711,30713],{"className":30712},[416],[63,30714,30716],{"className":30715,"style":28732},[420],[63,30717],{},[14,30719,30720],{},"通过等比数列求和化简，最终得到 GAE 的紧凑形式。",[14,30722,30723,30724,30776,30777,30828],{},"GAE 优势函数本质上是在 ",[63,30725,30727,30746],{"className":30726},[342],[63,30728,30730],{"className":30729},[346],[348,30731,30732],{"xmlns":350},[352,30733,30734,30743],{},[355,30735,30736,30738,30741],{},[371,30737,27920],{},[738,30739,30740],{},"→",[361,30742,1802],{},[378,30744,30745],{"encoding":380},"\\lambda \\to 0",[63,30747,30749,30767],{"className":30748,"ariaHidden":386},[385],[63,30750,30752,30755,30758,30761,30764],{"className":30751},[390],[63,30753],{"className":30754,"style":684},[394],[63,30756,27920],{"className":30757},[399,472],[63,30759],{"className":30760,"style":777},[776],[63,30762,30740],{"className":30763},[781],[63,30765],{"className":30766,"style":777},[776],[63,30768,30770,30773],{"className":30769},[390],[63,30771],{"className":30772,"style":1814},[394],[63,30774,1802],{"className":30775},[399],"（高偏差、低方差）与 ",[63,30778,30780,30798],{"className":30779},[342],[63,30781,30783],{"className":30782},[346],[348,30784,30785],{"xmlns":350},[352,30786,30787,30795],{},[355,30788,30789,30791,30793],{},[371,30790,27920],{},[738,30792,30740],{},[361,30794,363],{},[378,30796,30797],{"encoding":380},"\\lambda \\to 1",[63,30799,30801,30819],{"className":30800,"ariaHidden":386},[385],[63,30802,30804,30807,30810,30813,30816],{"className":30803},[390],[63,30805],{"className":30806,"style":684},[394],[63,30808,27920],{"className":30809},[399,472],[63,30811],{"className":30812,"style":777},[776],[63,30814,30740],{"className":30815},[781],[63,30817],{"className":30818,"style":777},[776],[63,30820,30822,30825],{"className":30821},[390],[63,30823],{"className":30824,"style":1814},[394],[63,30826,363],{"className":30827},[399],"（低偏差、高方差）之间做插值，从而平衡 bias 与 variance。",[14,30830,30831],{},"整理一下，到这里我们得到了三个关键表达式：",[63,30833,30835],{"className":30834},[14495],[63,30836,30838,31094],{"className":30837},[342],[63,30839,30841],{"className":30840},[346],[348,30842,30843],{"xmlns":350,"display":9561},[352,30844,30845,31091],{},[355,30846,30847,30855,30857,30863,30865,30867,30873,30875,30887,30889,30891,30897,30899,30905,30907,30909,30923,30925,30931,30933,30935,30937,30939,30953,30955,30957,30959,30965,30979,30981,30987,31001,31019,31033,31035,31043,31045,31053,31055,31057,31059,31061,31063,31069,31071,31079,31081,31089],{},[1870,30848,30849,30851,30853],{},[371,30850,25861],{},[371,30852,1057],{},[371,30854,949],{},[738,30856,204],{},[368,30858,30859,30861],{},[371,30860,7852],{},[371,30862,1057],{},[738,30864,5839],{},[371,30866,19142],{},[368,30868,30869,30871],{},[371,30870,949],{},[371,30872,14101],{},[738,30874,1078],{"stretchy":1077},[368,30876,30877,30879],{},[371,30878,12890],{},[355,30880,30881,30883,30885],{},[371,30882,1057],{},[738,30884,5839],{},[361,30886,363],{},[738,30888,1092],{"stretchy":1077},[738,30890,18511],{},[368,30892,30893,30895],{},[371,30894,949],{},[371,30896,14101],{},[738,30898,1078],{"stretchy":1077},[368,30900,30901,30903],{},[371,30902,12890],{},[371,30904,1057],{},[738,30906,1092],{"stretchy":1077},[776,30908],{"linebreak":21897},[1870,30910,30911,30913,30915],{},[371,30912,1054],{},[371,30914,14101],{},[355,30916,30917,30919,30921],{},[371,30918,27970],{},[371,30920,1054],{},[371,30922,5688],{},[738,30924,1078],{"stretchy":1077},[368,30926,30927,30929],{},[371,30928,12890],{},[371,30930,1057],{},[738,30932,1083],{"separator":386},[371,30934,38],{},[738,30936,1092],{"stretchy":1077},[738,30938,204],{},[14828,30940,30941,30943,30951],{},[738,30942,1874],{},[355,30944,30945,30947,30949],{},[371,30946,7380],{},[738,30948,204],{},[361,30950,1802],{},[371,30952,28503],{"mathvariant":1053},[738,30954,1078],{"stretchy":1077},[371,30956,19142],{},[371,30958,27920],{},[742,30960,30961,30963],{},[738,30962,1092],{"stretchy":1077},[371,30964,7380],{},[1870,30966,30967,30969,30977],{},[371,30968,25861],{},[355,30970,30971,30973,30975],{},[371,30972,1057],{},[738,30974,5839],{},[371,30976,7380],{},[371,30978,949],{},[776,30980],{"linebreak":21897},[358,30982,30983,30985],{},[361,30984,363],{},[371,30986,7342],{},[14828,30988,30989,30991,30999],{},[738,30990,1874],{},[355,30992,30993,30995,30997],{},[371,30994,612],{},[738,30996,204],{},[361,30998,363],{},[371,31000,7342],{},[14828,31002,31003,31005,31013],{},[738,31004,1874],{},[355,31006,31007,31009,31011],{},[371,31008,1057],{},[738,31010,204],{},[361,31012,363],{},[368,31014,31015,31017],{},[371,31016,13358],{},[371,31018,612],{},[1870,31020,31021,31023,31025],{},[371,31022,1054],{},[371,31024,14101],{},[355,31026,31027,31029,31031],{},[371,31028,27970],{},[371,31030,1054],{},[371,31032,5688],{},[738,31034,1078],{"stretchy":1077},[1870,31036,31037,31039,31041],{},[371,31038,12890],{},[371,31040,612],{},[371,31042,1057],{},[738,31044,1083],{"separator":386},[1870,31046,31047,31049,31051],{},[371,31048,38],{},[371,31050,612],{},[371,31052,1057],{},[738,31054,1092],{"stretchy":1077},[3343,31056,7365],{},[371,31058,14525],{"mathvariant":1053},[371,31060,14950],{},[738,31062,1100],{},[368,31064,31065,31067],{},[371,31066,5685],{},[371,31068,14101],{},[738,31070,1078],{"stretchy":1077},[1870,31072,31073,31075,31077],{},[371,31074,38],{},[371,31076,612],{},[371,31078,1057],{},[371,31080,13030],{"mathvariant":1053},[1870,31082,31083,31085,31087],{},[371,31084,12890],{},[371,31086,612],{},[371,31088,1057],{},[738,31090,1092],{"stretchy":1077},[378,31092,31093],{"encoding":380},"\\delta_t^V = r_t + \\gamma V_\\theta(s_{t+1}) - V_\\theta(s_t) \\\\\nA_\\theta^{GAE}(s_t, a) = \\sum_{b=0}^{\\infty} (\\gamma\\lambda)^b \\delta_{t+b}^V \\\\\n\\frac{1}{N}\\sum_{n=1}^N\\sum_{t=1}^{T_n} A_\\theta^{GAE}(s_n^t, a_n^t)\\, \\nabla \\log P_\\theta(a_n^t|s_n^t)",[63,31095,31097,31163,31218,31331,31423,31426,31556,31728,31731],{"className":31096,"ariaHidden":386},[385],[63,31098,31100,31103,31154,31157,31160],{"className":31099},[390],[63,31101],{"className":31102,"style":26306},[394],[63,31104,31106,31109],{"className":31105},[399],[63,31107,25861],{"className":31108,"style":25885},[399,472],[63,31110,31112],{"className":31111},[476],[63,31113,31115,31146],{"className":31114},[411,412],[63,31116,31118,31143],{"className":31117},[416],[63,31119,31121,31132],{"className":31120,"style":25181},[420],[63,31122,31123,31126],{"style":25900},[63,31124],{"className":31125,"style":817},[428],[63,31127,31129],{"className":31128},[433,434,435,436],[63,31130,1057],{"className":31131},[399,472,436],[63,31133,31134,31137],{"style":16572},[63,31135],{"className":31136,"style":817},[428],[63,31138,31140],{"className":31139},[433,434,435,436],[63,31141,949],{"className":31142,"style":979},[399,472,436],[63,31144,507],{"className":31145},[506],[63,31147,31149],{"className":31148},[416],[63,31150,31152],{"className":31151,"style":17487},[420],[63,31153],{},[63,31155],{"className":31156,"style":777},[776],[63,31158,204],{"className":31159},[781],[63,31161],{"className":31162,"style":777},[776],[63,31164,31166,31169,31209,31212,31215],{"className":31165},[390],[63,31167],{"className":31168,"style":22156},[394],[63,31170,31172,31175],{"className":31171},[399],[63,31173,7852],{"className":31174,"style":3276},[399,472],[63,31176,31178],{"className":31177},[476],[63,31179,31181,31201],{"className":31180},[411,412],[63,31182,31184,31198],{"className":31183},[416],[63,31185,31187],{"className":31186,"style":12818},[420],[63,31188,31189,31192],{"style":13208},[63,31190],{"className":31191,"style":817},[428],[63,31193,31195],{"className":31194},[433,434,435,436],[63,31196,1057],{"className":31197},[399,472,436],[63,31199,507],{"className":31200},[506],[63,31202,31204],{"className":31203},[416],[63,31205,31207],{"className":31206,"style":1554},[420],[63,31208],{},[63,31210],{"className":31211,"style":1920},[776],[63,31213,5839],{"className":31214},[830],[63,31216],{"className":31217,"style":1920},[776],[63,31219,31221,31224,31227,31267,31270,31319,31322,31325,31328],{"className":31220},[390],[63,31222],{"className":31223,"style":1155},[394],[63,31225,19142],{"className":31226,"style":19389},[399,472],[63,31228,31230,31233],{"className":31229},[399],[63,31231,949],{"className":31232,"style":979},[399,472],[63,31234,31236],{"className":31235},[476],[63,31237,31239,31259],{"className":31238},[411,412],[63,31240,31242,31256],{"className":31241},[416],[63,31243,31245],{"className":31244,"style":1532},[420],[63,31246,31247,31250],{"style":20739},[63,31248],{"className":31249,"style":817},[428],[63,31251,31253],{"className":31252},[433,434,435,436],[63,31254,14101],{"className":31255,"style":3276},[399,472,436],[63,31257,507],{"className":31258},[506],[63,31260,31262],{"className":31261},[416],[63,31263,31265],{"className":31264,"style":1554},[420],[63,31266],{},[63,31268,1078],{"className":31269},[403],[63,31271,31273,31276],{"className":31272},[399],[63,31274,12890],{"className":31275},[399,472],[63,31277,31279],{"className":31278},[476],[63,31280,31282,31311],{"className":31281},[411,412],[63,31283,31285,31308],{"className":31284},[416],[63,31286,31288],{"className":31287,"style":3157},[420],[63,31289,31290,31293],{"style":1535},[63,31291],{"className":31292,"style":817},[428],[63,31294,31296],{"className":31295},[433,434,435,436],[63,31297,31299,31302,31305],{"className":31298},[399,436],[63,31300,1057],{"className":31301},[399,472,436],[63,31303,5839],{"className":31304},[830,436],[63,31306,363],{"className":31307},[399,436],[63,31309,507],{"className":31310},[506],[63,31312,31314],{"className":31313},[416],[63,31315,31317],{"className":31316,"style":6959},[420],[63,31318],{},[63,31320,1092],{"className":31321},[593],[63,31323],{"className":31324,"style":1920},[776],[63,31326,18511],{"className":31327},[830],[63,31329],{"className":31330,"style":1920},[776],[63,31332,31334,31337,31377,31380,31420],{"className":31333},[390],[63,31335],{"className":31336,"style":1155},[394],[63,31338,31340,31343],{"className":31339},[399],[63,31341,949],{"className":31342,"style":979},[399,472],[63,31344,31346],{"className":31345},[476],[63,31347,31349,31369],{"className":31348},[411,412],[63,31350,31352,31366],{"className":31351},[416],[63,31353,31355],{"className":31354,"style":1532},[420],[63,31356,31357,31360],{"style":20739},[63,31358],{"className":31359,"style":817},[428],[63,31361,31363],{"className":31362},[433,434,435,436],[63,31364,14101],{"className":31365,"style":3276},[399,472,436],[63,31367,507],{"className":31368},[506],[63,31370,31372],{"className":31371},[416],[63,31373,31375],{"className":31374,"style":1554},[420],[63,31376],{},[63,31378,1078],{"className":31379},[403],[63,31381,31383,31386],{"className":31382},[399],[63,31384,12890],{"className":31385},[399,472],[63,31387,31389],{"className":31388},[476],[63,31390,31392,31412],{"className":31391},[411,412],[63,31393,31395,31409],{"className":31394},[416],[63,31396,31398],{"className":31397,"style":12818},[420],[63,31399,31400,31403],{"style":1535},[63,31401],{"className":31402,"style":817},[428],[63,31404,31406],{"className":31405},[433,434,435,436],[63,31407,1057],{"className":31408},[399,472,436],[63,31410,507],{"className":31411},[506],[63,31413,31415],{"className":31414},[416],[63,31416,31418],{"className":31417,"style":1554},[420],[63,31419],{},[63,31421,1092],{"className":31422},[593],[63,31424],{"className":31425},[776,21897],[63,31427,31429,31432,31492,31495,31535,31538,31541,31544,31547,31550,31553],{"className":31428},[390],[63,31430],{"className":31431,"style":25162},[394],[63,31433,31435,31438],{"className":31434},[399],[63,31436,1054],{"className":31437},[399,472],[63,31439,31441],{"className":31440},[476],[63,31442,31444,31484],{"className":31443},[411,412],[63,31445,31447,31481],{"className":31446},[416],[63,31448,31450,31461],{"className":31449,"style":25181},[420],[63,31451,31452,31455],{"style":17457},[63,31453],{"className":31454,"style":817},[428],[63,31456,31458],{"className":31457},[433,434,435,436],[63,31459,14101],{"className":31460,"style":3276},[399,472,436],[63,31462,31463,31466],{"style":16572},[63,31464],{"className":31465,"style":817},[428],[63,31467,31469],{"className":31468},[433,434,435,436],[63,31470,31472,31475,31478],{"className":31471},[399,436],[63,31473,27970],{"className":31474},[399,472,436],[63,31476,1054],{"className":31477},[399,472,436],[63,31479,5688],{"className":31480,"style":6101},[399,472,436],[63,31482,507],{"className":31483},[506],[63,31485,31487],{"className":31486},[416],[63,31488,31490],{"className":31489,"style":17487},[420],[63,31491],{},[63,31493,1078],{"className":31494},[403],[63,31496,31498,31501],{"className":31497},[399],[63,31499,12890],{"className":31500},[399,472],[63,31502,31504],{"className":31503},[476],[63,31505,31507,31527],{"className":31506},[411,412],[63,31508,31510,31524],{"className":31509},[416],[63,31511,31513],{"className":31512,"style":12818},[420],[63,31514,31515,31518],{"style":1535},[63,31516],{"className":31517,"style":817},[428],[63,31519,31521],{"className":31520},[433,434,435,436],[63,31522,1057],{"className":31523},[399,472,436],[63,31525,507],{"className":31526},[506],[63,31528,31530],{"className":31529},[416],[63,31531,31533],{"className":31532,"style":1554},[420],[63,31534],{},[63,31536,1083],{"className":31537},[1173],[63,31539],{"className":31540,"style":1177},[776],[63,31542,38],{"className":31543},[399,472],[63,31545,1092],{"className":31546},[593],[63,31548],{"className":31549,"style":777},[776],[63,31551,204],{"className":31552},[781],[63,31554],{"className":31555,"style":777},[776],[63,31557,31559,31563,31630,31633,31636,31639,31668],{"className":31558},[390],[63,31560],{"className":31561,"style":31562},[394],"height:2.9535em;vertical-align:-1.3021em;",[63,31564,31566],{"className":31565},[1212,14383],[63,31567,31569,31622],{"className":31568},[411,412],[63,31570,31572,31619],{"className":31571},[416],[63,31573,31575,31595,31605],{"className":31574,"style":30553},[420],[63,31576,31577,31580],{"style":30556},[63,31578],{"className":31579,"style":15577},[428],[63,31581,31583],{"className":31582},[433,434,435,436],[63,31584,31586,31589,31592],{"className":31585},[399,436],[63,31587,7380],{"className":31588},[399,472,436],[63,31590,204],{"className":31591},[781,436],[63,31593,1802],{"className":31594},[399,436],[63,31596,31597,31600],{"style":15586},[63,31598],{"className":31599,"style":15577},[428],[63,31601,31602],{},[63,31603,1874],{"className":31604},[1212,1958,15595],[63,31606,31607,31610],{"style":16520},[63,31608],{"className":31609,"style":15577},[428],[63,31611,31613],{"className":31612},[433,434,435,436],[63,31614,31616],{"className":31615},[399,436],[63,31617,28503],{"className":31618},[399,436],[63,31620,507],{"className":31621},[506],[63,31623,31625],{"className":31624},[416],[63,31626,31628],{"className":31627,"style":30605},[420],[63,31629],{},[63,31631,1078],{"className":31632},[403],[63,31634,19142],{"className":31635,"style":19389},[399,472],[63,31637,27920],{"className":31638},[399,472],[63,31640,31642,31645],{"className":31641},[593],[63,31643,1092],{"className":31644},[593],[63,31646,31648],{"className":31647},[476],[63,31649,31651],{"className":31650},[411],[63,31652,31654],{"className":31653},[416],[63,31655,31657],{"className":31656,"style":30635},[420],[63,31658,31659,31662],{"style":16572},[63,31660],{"className":31661,"style":817},[428],[63,31663,31665],{"className":31664},[433,434,435,436],[63,31666,7380],{"className":31667},[399,472,436],[63,31669,31671,31674],{"className":31670},[399],[63,31672,25861],{"className":31673,"style":25885},[399,472],[63,31675,31677],{"className":31676},[476],[63,31678,31680,31720],{"className":31679},[411,412],[63,31681,31683,31717],{"className":31682},[416],[63,31684,31686,31706],{"className":31685,"style":25181},[420],[63,31687,31688,31691],{"style":25900},[63,31689],{"className":31690,"style":817},[428],[63,31692,31694],{"className":31693},[433,434,435,436],[63,31695,31697,31700,31703],{"className":31696},[399,436],[63,31698,1057],{"className":31699},[399,472,436],[63,31701,5839],{"className":31702},[830,436],[63,31704,7380],{"className":31705},[399,472,436],[63,31707,31708,31711],{"style":16572},[63,31709],{"className":31710,"style":817},[428],[63,31712,31714],{"className":31713},[433,434,435,436],[63,31715,949],{"className":31716,"style":979},[399,472,436],[63,31718,507],{"className":31719},[506],[63,31721,31723],{"className":31722},[416],[63,31724,31726],{"className":31725,"style":26710},[420],[63,31727],{},[63,31729],{"className":31730},[776,21897],[63,31732,31734,31738,31800,31803,31867,31870,31974,31977,32037,32040,32091,32094,32097,32148,32151,32154,32157,32160,32165,32168,32208,32211,32262,32265,32316],{"className":31733},[390],[63,31735],{"className":31736,"style":31737},[394],"height:3.1065em;vertical-align:-1.2671em;",[63,31739,31741,31744,31797],{"className":31740},[399],[63,31742],{"className":31743},[403,404],[63,31745,31747],{"className":31746},[358],[63,31748,31750,31789],{"className":31749},[411,412],[63,31751,31753,31786],{"className":31752},[416],[63,31754,31756,31767,31775],{"className":31755,"style":16424},[420],[63,31757,31758,31761],{"style":15964},[63,31759],{"className":31760,"style":429},[428],[63,31762,31764],{"className":31763},[399],[63,31765,7342],{"className":31766,"style":16436},[399,472],[63,31768,31769,31772],{"style":554},[63,31770],{"className":31771,"style":429},[428],[63,31773],{"className":31774,"style":562},[561],[63,31776,31777,31780],{"style":16030},[63,31778],{"className":31779,"style":429},[428],[63,31781,31783],{"className":31782},[399],[63,31784,363],{"className":31785},[399],[63,31787,507],{"className":31788},[506],[63,31790,31792],{"className":31791},[416],[63,31793,31795],{"className":31794,"style":16465},[420],[63,31796],{},[63,31798],{"className":31799},[593,404],[63,31801],{"className":31802,"style":1177},[776],[63,31804,31806],{"className":31805},[1212,14383],[63,31807,31809,31859],{"className":31808},[411,412],[63,31810,31812,31856],{"className":31811},[416],[63,31813,31815,31835,31845],{"className":31814,"style":16486},[420],[63,31816,31817,31820],{"style":16489},[63,31818],{"className":31819,"style":15577},[428],[63,31821,31823],{"className":31822},[433,434,435,436],[63,31824,31826,31829,31832],{"className":31825},[399,436],[63,31827,612],{"className":31828},[399,472,436],[63,31830,204],{"className":31831},[781,436],[63,31833,363],{"className":31834},[399,436],[63,31836,31837,31840],{"style":15586},[63,31838],{"className":31839,"style":15577},[428],[63,31841,31842],{},[63,31843,1874],{"className":31844},[1212,1958,15595],[63,31846,31847,31850],{"style":16520},[63,31848],{"className":31849,"style":15577},[428],[63,31851,31853],{"className":31852},[433,434,435,436],[63,31854,7342],{"className":31855,"style":16436},[399,472,436],[63,31857,507],{"className":31858},[506],[63,31860,31862],{"className":31861},[416],[63,31863,31865],{"className":31864,"style":16539},[420],[63,31866],{},[63,31868],{"className":31869,"style":1177},[776],[63,31871,31873],{"className":31872},[1212,14383],[63,31874,31876,31966],{"className":31875},[411,412],[63,31877,31879,31963],{"className":31878},[416],[63,31880,31882,31902,31912],{"className":31881,"style":17295},[420],[63,31883,31884,31887],{"style":16489},[63,31885],{"className":31886,"style":15577},[428],[63,31888,31890],{"className":31889},[433,434,435,436],[63,31891,31893,31896,31899],{"className":31892},[399,436],[63,31894,1057],{"className":31895},[399,472,436],[63,31897,204],{"className":31898},[781,436],[63,31900,363],{"className":31901},[399,436],[63,31903,31904,31907],{"style":15586},[63,31905],{"className":31906,"style":15577},[428],[63,31908,31909],{},[63,31910,1874],{"className":31911},[1212,1958,15595],[63,31913,31914,31917],{"style":17328},[63,31915],{"className":31916,"style":15577},[428],[63,31918,31920],{"className":31919},[433,434,435,436],[63,31921,31923],{"className":31922},[399,436],[63,31924,31926,31929],{"className":31925},[399,436],[63,31927,13358],{"className":31928,"style":3251},[399,472,436],[63,31930,31932],{"className":31931},[476],[63,31933,31935,31955],{"className":31934},[411,412],[63,31936,31938,31952],{"className":31937},[416],[63,31939,31941],{"className":31940,"style":4367},[420],[63,31942,31943,31946],{"style":17358},[63,31944],{"className":31945,"style":493},[428],[63,31947,31949],{"className":31948},[433,497,498,436],[63,31950,612],{"className":31951},[399,472,436],[63,31953,507],{"className":31954},[506],[63,31956,31958],{"className":31957},[416],[63,31959,31961],{"className":31960,"style":4389},[420],[63,31962],{},[63,31964,507],{"className":31965},[506],[63,31967,31969],{"className":31968},[416],[63,31970,31972],{"className":31971,"style":16539},[420],[63,31973],{},[63,31975],{"className":31976,"style":1177},[776],[63,31978,31980,31983],{"className":31979},[399],[63,31981,1054],{"className":31982},[399,472],[63,31984,31986],{"className":31985},[476],[63,31987,31989,32029],{"className":31988},[411,412],[63,31990,31992,32026],{"className":31991},[416],[63,31993,31995,32006],{"className":31994,"style":25181},[420],[63,31996,31997,32000],{"style":17457},[63,31998],{"className":31999,"style":817},[428],[63,32001,32003],{"className":32002},[433,434,435,436],[63,32004,14101],{"className":32005,"style":3276},[399,472,436],[63,32007,32008,32011],{"style":16572},[63,32009],{"className":32010,"style":817},[428],[63,32012,32014],{"className":32013},[433,434,435,436],[63,32015,32017,32020,32023],{"className":32016},[399,436],[63,32018,27970],{"className":32019},[399,472,436],[63,32021,1054],{"className":32022},[399,472,436],[63,32024,5688],{"className":32025,"style":6101},[399,472,436],[63,32027,507],{"className":32028},[506],[63,32030,32032],{"className":32031},[416],[63,32033,32035],{"className":32034,"style":17487},[420],[63,32036],{},[63,32038,1078],{"className":32039},[403],[63,32041,32043,32046],{"className":32042},[399],[63,32044,12890],{"className":32045},[399,472],[63,32047,32049],{"className":32048},[476],[63,32050,32052,32083],{"className":32051},[411,412],[63,32053,32055,32080],{"className":32054},[416],[63,32056,32058,32069],{"className":32057,"style":17454},[420],[63,32059,32060,32063],{"style":17457},[63,32061],{"className":32062,"style":817},[428],[63,32064,32066],{"className":32065},[433,434,435,436],[63,32067,612],{"className":32068},[399,472,436],[63,32070,32071,32074],{"style":16572},[63,32072],{"className":32073,"style":817},[428],[63,32075,32077],{"className":32076},[433,434,435,436],[63,32078,1057],{"className":32079},[399,472,436],[63,32081,507],{"className":32082},[506],[63,32084,32086],{"className":32085},[416],[63,32087,32089],{"className":32088,"style":17487},[420],[63,32090],{},[63,32092,1083],{"className":32093},[1173],[63,32095],{"className":32096,"style":1177},[776],[63,32098,32100,32103],{"className":32099},[399],[63,32101,38],{"className":32102},[399,472],[63,32104,32106],{"className":32105},[476],[63,32107,32109,32140],{"className":32108},[411,412],[63,32110,32112,32137],{"className":32111},[416],[63,32113,32115,32126],{"className":32114,"style":17454},[420],[63,32116,32117,32120],{"style":17457},[63,32118],{"className":32119,"style":817},[428],[63,32121,32123],{"className":32122},[433,434,435,436],[63,32124,612],{"className":32125},[399,472,436],[63,32127,32128,32131],{"style":16572},[63,32129],{"className":32130,"style":817},[428],[63,32132,32134],{"className":32133},[433,434,435,436],[63,32135,1057],{"className":32136},[399,472,436],[63,32138,507],{"className":32139},[506],[63,32141,32143],{"className":32142},[416],[63,32144,32146],{"className":32145,"style":17487},[420],[63,32147],{},[63,32149,1092],{"className":32150},[593],[63,32152],{"className":32153,"style":1177},[776],[63,32155,14525],{"className":32156},[399],[63,32158],{"className":32159,"style":1177},[776],[63,32161,16996,32163],{"className":32162},[1212],[63,32164,17000],{"style":16999},[63,32166],{"className":32167,"style":1177},[776],[63,32169,32171,32174],{"className":32170},[399],[63,32172,5685],{"className":32173,"style":3251},[399,472],[63,32175,32177],{"className":32176},[476],[63,32178,32180,32200],{"className":32179},[411,412],[63,32181,32183,32197],{"className":32182},[416],[63,32184,32186],{"className":32185,"style":1532},[420],[63,32187,32188,32191],{"style":7483},[63,32189],{"className":32190,"style":817},[428],[63,32192,32194],{"className":32193},[433,434,435,436],[63,32195,14101],{"className":32196,"style":3276},[399,472,436],[63,32198,507],{"className":32199},[506],[63,32201,32203],{"className":32202},[416],[63,32204,32206],{"className":32205,"style":1554},[420],[63,32207],{},[63,32209,1078],{"className":32210},[403],[63,32212,32214,32217],{"className":32213},[399],[63,32215,38],{"className":32216},[399,472],[63,32218,32220],{"className":32219},[476],[63,32221,32223,32254],{"className":32222},[411,412],[63,32224,32226,32251],{"className":32225},[416],[63,32227,32229,32240],{"className":32228,"style":17454},[420],[63,32230,32231,32234],{"style":17457},[63,32232],{"className":32233,"style":817},[428],[63,32235,32237],{"className":32236},[433,434,435,436],[63,32238,612],{"className":32239},[399,472,436],[63,32241,32242,32245],{"style":16572},[63,32243],{"className":32244,"style":817},[428],[63,32246,32248],{"className":32247},[433,434,435,436],[63,32249,1057],{"className":32250},[399,472,436],[63,32252,507],{"className":32253},[506],[63,32255,32257],{"className":32256},[416],[63,32258,32260],{"className":32259,"style":17487},[420],[63,32261],{},[63,32263,13030],{"className":32264},[399],[63,32266,32268,32271],{"className":32267},[399],[63,32269,12890],{"className":32270},[399,472],[63,32272,32274],{"className":32273},[476],[63,32275,32277,32308],{"className":32276},[411,412],[63,32278,32280,32305],{"className":32279},[416],[63,32281,32283,32294],{"className":32282,"style":17454},[420],[63,32284,32285,32288],{"style":17457},[63,32286],{"className":32287,"style":817},[428],[63,32289,32291],{"className":32290},[433,434,435,436],[63,32292,612],{"className":32293},[399,472,436],[63,32295,32296,32299],{"style":16572},[63,32297],{"className":32298,"style":817},[428],[63,32300,32302],{"className":32301},[433,434,435,436],[63,32303,1057],{"className":32304},[399,472,436],[63,32306,507],{"className":32307},[506],[63,32309,32311],{"className":32310},[416],[63,32312,32314],{"className":32313,"style":17487},[420],[63,32315],{},[63,32317,1092],{"className":32318},[593],[14,32320,32321,32322,32391],{},"这里的状态价值函数 ",[63,32323,32325,32342],{"className":32324},[342],[63,32326,32328],{"className":32327},[346],[348,32329,32330],{"xmlns":350},[352,32331,32332,32340],{},[355,32333,32334],{},[368,32335,32336,32338],{},[371,32337,949],{},[371,32339,14101],{},[378,32341,23087],{"encoding":380},[63,32343,32345],{"className":32344,"ariaHidden":386},[385],[63,32346,32348,32351],{"className":32347},[390],[63,32349],{"className":32350,"style":6995},[394],[63,32352,32354,32357],{"className":32353},[399],[63,32355,949],{"className":32356,"style":979},[399,472],[63,32358,32360],{"className":32359},[476],[63,32361,32363,32383],{"className":32362},[411,412],[63,32364,32366,32380],{"className":32365},[416],[63,32367,32369],{"className":32368,"style":1532},[420],[63,32370,32371,32374],{"style":20739},[63,32372],{"className":32373,"style":817},[428],[63,32375,32377],{"className":32376},[433,434,435,436],[63,32378,14101],{"className":32379,"style":3276},[399,472,436],[63,32381,507],{"className":32382},[506],[63,32384,32386],{"className":32385},[416],[63,32387,32389],{"className":32388,"style":1554},[420],[63,32390],{}," 一般用一个神经网络来拟合（即 critic 网络），可以与策略网络共用主干参数，仅在最后一层分叉为两个 head。",[10,32393,32395],{"id":32394},"proximal-policy-optimization-ppo-近端策略优化算法","Proximal Policy Optimization (PPO) 近端策略优化算法",[14,32397,32398,32399,32402,32403,32472,32473,20807,32476,32582,32583,32586,32587,235],{},"在经典的强化学习训练范式里，我们通常一边采集数据、一边更新模型，采过的数据用一次就丢掉——这种做法被称为 ",[11217,32400,32401],{},"on-policy","。问题在于，强化学习的环境交互成本往往很高，这样\"用一次就扔\"显然非常浪费。如果我们能让当前策略 ",[63,32404,32406,32423],{"className":32405},[342],[63,32407,32409],{"className":32408},[346],[348,32410,32411],{"xmlns":350},[352,32412,32413,32421],{},[355,32414,32415],{},[368,32416,32417,32419],{},[371,32418,6764],{},[371,32420,14101],{},[378,32422,14104],{"encoding":380},[63,32424,32426],{"className":32425,"ariaHidden":386},[385],[63,32427,32429,32432],{"className":32428},[390],[63,32430],{"className":32431,"style":12799},[394],[63,32433,32435,32438],{"className":32434},[399],[63,32436,6764],{"className":32437,"style":718},[399,472],[63,32439,32441],{"className":32440},[476],[63,32442,32444,32464],{"className":32443},[411,412],[63,32445,32447,32461],{"className":32446},[416],[63,32448,32450],{"className":32449,"style":1532},[420],[63,32451,32452,32455],{"style":2083},[63,32453],{"className":32454,"style":817},[428],[63,32456,32458],{"className":32457},[433,434,435,436],[63,32459,14101],{"className":32460,"style":3276},[399,472,436],[63,32462,507],{"className":32463},[506],[63,32465,32467],{"className":32466},[416],[63,32468,32470],{"className":32469,"style":1554},[420],[63,32471],{}," 复用 ",[11217,32474,32475],{},"旧策略",[63,32477,32479,32501],{"className":32478},[342],[63,32480,32482],{"className":32481},[346],[348,32483,32484],{"xmlns":350},[352,32485,32486,32498],{},[355,32487,32488],{},[368,32489,32490,32492],{},[371,32491,6764],{},[742,32493,32494,32496],{},[371,32495,14101],{},[738,32497,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[378,32499,32500],{"encoding":380},"\\pi_{\\theta'}",[63,32502,32504],{"className":32503,"ariaHidden":386},[385],[63,32505,32507,32510],{"className":32506},[390],[63,32508],{"className":32509,"style":12799},[394],[63,32511,32513,32516],{"className":32512},[399],[63,32514,6764],{"className":32515,"style":718},[399,472],[63,32517,32519],{"className":32518},[476],[63,32520,32522,32574],{"className":32521},[411,412],[63,32523,32525,32571],{"className":32524},[416],[63,32526,32528],{"className":32527,"style":1532},[420],[63,32529,32530,32533],{"style":2083},[63,32531],{"className":32532,"style":817},[428],[63,32534,32536],{"className":32535},[433,434,435,436],[63,32537,32539],{"className":32538},[399,436],[63,32540,32542,32545],{"className":32541},[399,436],[63,32543,14101],{"className":32544,"style":3276},[399,472,436],[63,32546,32548],{"className":32547},[476],[63,32549,32551],{"className":32550},[411],[63,32552,32554],{"className":32553},[416],[63,32555,32557],{"className":32556,"style":19291},[420],[63,32558,32559,32562],{"style":19294},[63,32560],{"className":32561,"style":493},[428],[63,32563,32565],{"className":32564},[433,497,498,436],[63,32566,32568],{"className":32567},[399,436],[63,32569,19131],{"className":32570},[399,436],[63,32572,507],{"className":32573},[506],[63,32575,32577],{"className":32576},[416],[63,32578,32580],{"className":32579,"style":1554},[420],[63,32581],{}," 采集的数据进行训练（即 ",[11217,32584,32585],{},"off-policy","），训练效率就能显著提升。要实现这一点，关键就是 ",[11217,32588,32589],{},"重要性采样（Importance Sampling）",[30,32591,32592],{"id":32592},"重要性采样",[14,32594,32595,32596,32640,32641,32669,32670,32843,32844,32872,32873,32901,32902,32930],{},"我们可以把\"",[63,32597,32599,32619],{"className":32598},[342],[63,32600,32602],{"className":32601},[346],[348,32603,32604],{"xmlns":350},[352,32605,32606,32616],{},[355,32607,32608,32610,32612,32614],{},[371,32609,11741],{},[738,32611,1078],{"stretchy":1077},[371,32613,7347],{},[738,32615,1092],{"stretchy":1077},[378,32617,32618],{"encoding":380},"f(x)",[63,32620,32622],{"className":32621,"ariaHidden":386},[385],[63,32623,32625,32628,32631,32634,32637],{"className":32624},[390],[63,32626],{"className":32627,"style":1155},[394],[63,32629,11741],{"className":32630,"style":13736},[399,472],[63,32632,1078],{"className":32633},[403],[63,32635,7347],{"className":32636},[399,472],[63,32638,1092],{"className":32639},[593]," 在分布 ",[63,32642,32644,32657],{"className":32643},[342],[63,32645,32647],{"className":32646},[346],[348,32648,32649],{"xmlns":350},[352,32650,32651,32655],{},[355,32652,32653],{},[371,32654,14],{},[378,32656,14],{"encoding":380},[63,32658,32660],{"className":32659,"ariaHidden":386},[385],[63,32661,32663,32666],{"className":32662},[390],[63,32664],{"className":32665,"style":1754},[394],[63,32667,14],{"className":32668},[399,472]," 下的期望\"改写为\"",[63,32671,32673,32717],{"className":32672},[342],[63,32674,32676],{"className":32675},[346],[348,32677,32678],{"xmlns":350},[352,32679,32680,32714],{},[355,32681,32682,32684,32686,32688,32690,32692],{},[371,32683,11741],{},[738,32685,1078],{"stretchy":1077},[371,32687,7347],{},[738,32689,1092],{"stretchy":1077},[738,32691,1864],{},[358,32693,32694,32704],{},[355,32695,32696,32698,32700,32702],{},[371,32697,14],{},[738,32699,1078],{"stretchy":1077},[371,32701,7347],{},[738,32703,1092],{"stretchy":1077},[355,32705,32706,32708,32710,32712],{},[371,32707,1742],{},[738,32709,1078],{"stretchy":1077},[371,32711,7347],{},[738,32713,1092],{"stretchy":1077},[378,32715,32716],{"encoding":380},"f(x) \\cdot \\frac{p(x)}{q(x)}",[63,32718,32720,32747],{"className":32719,"ariaHidden":386},[385],[63,32721,32723,32726,32729,32732,32735,32738,32741,32744],{"className":32722},[390],[63,32724],{"className":32725,"style":1155},[394],[63,32727,11741],{"className":32728,"style":13736},[399,472],[63,32730,1078],{"className":32731},[403],[63,32733,7347],{"className":32734},[399,472],[63,32736,1092],{"className":32737},[593],[63,32739],{"className":32740,"style":1920},[776],[63,32742,1864],{"className":32743},[830],[63,32745],{"className":32746,"style":1920},[776],[63,32748,32750,32754],{"className":32749},[390],[63,32751],{"className":32752,"style":32753},[394],"height:1.53em;vertical-align:-0.52em;",[63,32755,32757,32760,32840],{"className":32756},[399],[63,32758],{"className":32759},[403,404],[63,32761,32763],{"className":32762},[358],[63,32764,32766,32831],{"className":32765},[411,412],[63,32767,32769,32828],{"className":32768},[416],[63,32770,32773,32796,32804],{"className":32771,"style":32772},[420],"height:1.01em;",[63,32774,32775,32778],{"style":18653},[63,32776],{"className":32777,"style":429},[428],[63,32779,32781],{"className":32780},[433,434,435,436],[63,32782,32784,32787,32790,32793],{"className":32783},[399,436],[63,32785,1742],{"className":32786,"style":718},[399,472,436],[63,32788,1078],{"className":32789},[403,436],[63,32791,7347],{"className":32792},[399,472,436],[63,32794,1092],{"className":32795},[593,436],[63,32797,32798,32801],{"style":554},[63,32799],{"className":32800,"style":429},[428],[63,32802],{"className":32803,"style":562},[561],[63,32805,32807,32810],{"style":32806},"top:-3.485em;",[63,32808],{"className":32809,"style":429},[428],[63,32811,32813],{"className":32812},[433,434,435,436],[63,32814,32816,32819,32822,32825],{"className":32815},[399,436],[63,32817,14],{"className":32818},[399,472,436],[63,32820,1078],{"className":32821},[403,436],[63,32823,7347],{"className":32824},[399,472,436],[63,32826,1092],{"className":32827},[593,436],[63,32829,507],{"className":32830},[506],[63,32832,32834],{"className":32833},[416],[63,32835,32838],{"className":32836,"style":32837},[420],"height:0.52em;",[63,32839],{},[63,32841],{"className":32842},[593,404]," 在另一分布 ",[63,32845,32847,32860],{"className":32846},[342],[63,32848,32850],{"className":32849},[346],[348,32851,32852],{"xmlns":350},[352,32853,32854,32858],{},[355,32855,32856],{},[371,32857,1742],{},[378,32859,1742],{"encoding":380},[63,32861,32863],{"className":32862,"ariaHidden":386},[385],[63,32864,32866,32869],{"className":32865},[390],[63,32867],{"className":32868,"style":1754},[394],[63,32870,1742],{"className":32871,"style":718},[399,472],"（proposal distribution）下的期望\"，这样就能用从 ",[63,32874,32876,32889],{"className":32875},[342],[63,32877,32879],{"className":32878},[346],[348,32880,32881],{"xmlns":350},[352,32882,32883,32887],{},[355,32884,32885],{},[371,32886,1742],{},[378,32888,1742],{"encoding":380},[63,32890,32892],{"className":32891,"ariaHidden":386},[385],[63,32893,32895,32898],{"className":32894},[390],[63,32896],{"className":32897,"style":1754},[394],[63,32899,1742],{"className":32900,"style":718},[399,472]," 采样的数据来估计原本在 ",[63,32903,32905,32918],{"className":32904},[342],[63,32906,32908],{"className":32907},[346],[348,32909,32910],{"xmlns":350},[352,32911,32912,32916],{},[355,32913,32914],{},[371,32915,14],{},[378,32917,14],{"encoding":380},[63,32919,32921],{"className":32920,"ariaHidden":386},[385],[63,32922,32924,32927],{"className":32923},[390],[63,32925],{"className":32926,"style":1754},[394],[63,32928,14],{"className":32929},[399,472]," 下的期望：",[63,32932,32934],{"className":32933},[14495],[63,32935,32937,33321],{"className":32936},[342],[63,32938,32940],{"className":32939},[346],[348,32941,32942],{"xmlns":350,"display":9561},[352,32943,32944,33318],{},[14507,32945,32946,33018,33082,33146,33218],{"rowspacing":14509,"columnalign":14510,"columnspacing":14511},[14513,32947,32948,32984],{},[14516,32949,32950],{},[14519,32951,32952],{"scriptlevel":1802,"displaystyle":386},[355,32953,32954,32956,32958,32960,32962,32964,32966],{},[371,32955,5688],{"mathvariant":746},[738,32957,1078],{"stretchy":1077},[371,32959,11741],{},[738,32961,1078],{"stretchy":1077},[371,32963,7347],{},[738,32965,1092],{"stretchy":1077},[368,32967,32968,32970],{},[738,32969,1092],{"stretchy":1077},[355,32971,32972,32974,32976,32978,32980,32982],{},[371,32973,7347],{},[738,32975,13856],{},[371,32977,14],{},[738,32979,1078],{"stretchy":1077},[371,32981,7347],{},[738,32983,1092],{"stretchy":1077},[14516,32985,32986],{},[14519,32987,32988],{"scriptlevel":1802,"displaystyle":386},[355,32989,32990,32992,32994,33000,33002,33004,33006,33008,33010,33012,33014,33016],{},[355,32991],{},[738,32993,204],{},[14206,32995,32996,32998],{},[738,32997,1874],{},[371,32999,7347],{},[371,33001,11741],{},[738,33003,1078],{"stretchy":1077},[371,33005,7347],{},[738,33007,1092],{"stretchy":1077},[738,33009,1864],{},[371,33011,14],{},[738,33013,1078],{"stretchy":1077},[371,33015,7347],{},[738,33017,1092],{"stretchy":1077},[14513,33019,33020,33026],{},[14516,33021,33022],{},[14519,33023,33024],{"scriptlevel":1802,"displaystyle":386},[355,33025],{},[14516,33027,33028],{},[14519,33029,33030],{"scriptlevel":1802,"displaystyle":386},[355,33031,33032,33034,33036,33042,33044,33046,33048,33050,33052,33054,33056,33058,33060],{},[355,33033],{},[738,33035,204],{},[14206,33037,33038,33040],{},[738,33039,1874],{},[371,33041,7347],{},[371,33043,11741],{},[738,33045,1078],{"stretchy":1077},[371,33047,7347],{},[738,33049,1092],{"stretchy":1077},[738,33051,1864],{},[371,33053,14],{},[738,33055,1078],{"stretchy":1077},[371,33057,7347],{},[738,33059,1092],{"stretchy":1077},[358,33061,33062,33072],{},[355,33063,33064,33066,33068,33070],{},[371,33065,1742],{},[738,33067,1078],{"stretchy":1077},[371,33069,7347],{},[738,33071,1092],{"stretchy":1077},[355,33073,33074,33076,33078,33080],{},[371,33075,1742],{},[738,33077,1078],{"stretchy":1077},[371,33079,7347],{},[738,33081,1092],{"stretchy":1077},[14513,33083,33084,33090],{},[14516,33085,33086],{},[14519,33087,33088],{"scriptlevel":1802,"displaystyle":386},[355,33089],{},[14516,33091,33092],{},[14519,33093,33094],{"scriptlevel":1802,"displaystyle":386},[355,33095,33096,33098,33100,33106,33108,33110,33112,33114,33136,33138,33140,33142,33144],{},[355,33097],{},[738,33099,204],{},[14206,33101,33102,33104],{},[738,33103,1874],{},[371,33105,7347],{},[371,33107,11741],{},[738,33109,1078],{"stretchy":1077},[371,33111,7347],{},[738,33113,1092],{"stretchy":1077},[358,33115,33116,33126],{},[355,33117,33118,33120,33122,33124],{},[371,33119,14],{},[738,33121,1078],{"stretchy":1077},[371,33123,7347],{},[738,33125,1092],{"stretchy":1077},[355,33127,33128,33130,33132,33134],{},[371,33129,1742],{},[738,33131,1078],{"stretchy":1077},[371,33133,7347],{},[738,33135,1092],{"stretchy":1077},[738,33137,1864],{},[371,33139,1742],{},[738,33141,1078],{"stretchy":1077},[371,33143,7347],{},[738,33145,1092],{"stretchy":1077},[14513,33147,33148,33154],{},[14516,33149,33150],{},[14519,33151,33152],{"scriptlevel":1802,"displaystyle":386},[355,33153],{},[14516,33155,33156],{},[14519,33157,33158],{"scriptlevel":1802,"displaystyle":386},[355,33159,33160,33162,33164,33166],{},[355,33161],{},[738,33163,204],{},[371,33165,5688],{"mathvariant":746},[368,33167,33168,33204],{},[355,33169,33170,33172,33174,33176,33178,33180,33202],{},[738,33171,1078],{"fence":386},[371,33173,11741],{},[738,33175,1078],{"stretchy":1077},[371,33177,7347],{},[738,33179,1092],{"stretchy":1077},[358,33181,33182,33192],{},[355,33183,33184,33186,33188,33190],{},[371,33185,14],{},[738,33187,1078],{"stretchy":1077},[371,33189,7347],{},[738,33191,1092],{"stretchy":1077},[355,33193,33194,33196,33198,33200],{},[371,33195,1742],{},[738,33197,1078],{"stretchy":1077},[371,33199,7347],{},[738,33201,1092],{"stretchy":1077},[738,33203,1092],{"fence":386},[355,33205,33206,33208,33210,33212,33214,33216],{},[371,33207,7347],{},[738,33209,13856],{},[371,33211,1742],{},[738,33213,1078],{"stretchy":1077},[371,33215,7347],{},[738,33217,1092],{"stretchy":1077},[14513,33219,33220,33226],{},[14516,33221,33222],{},[14519,33223,33224],{"scriptlevel":1802,"displaystyle":386},[355,33225],{},[14516,33227,33228],{},[14519,33229,33230],{"scriptlevel":1802,"displaystyle":386},[355,33231,33232,33234,33236,33242,33256,33258,33260,33266,33268,33298,33300,33302,33308,33310,33312,33314,33316],{},[355,33233],{},[738,33235,14820],{},[358,33237,33238,33240],{},[361,33239,363],{},[371,33241,7342],{},[14828,33243,33244,33246,33254],{},[738,33245,1874],{},[355,33247,33248,33250,33252],{},[371,33249,612],{},[738,33251,204],{},[361,33253,363],{},[371,33255,7342],{},[371,33257,11741],{},[738,33259,1078],{"stretchy":1077},[368,33261,33262,33264],{},[371,33263,7347],{},[371,33265,612],{},[738,33267,1092],{"stretchy":1077},[358,33269,33270,33284],{},[355,33271,33272,33274,33276,33282],{},[371,33273,14],{},[738,33275,1078],{"stretchy":1077},[368,33277,33278,33280],{},[371,33279,7347],{},[371,33281,612],{},[738,33283,1092],{"stretchy":1077},[355,33285,33286,33288,33290,33296],{},[371,33287,1742],{},[738,33289,1078],{"stretchy":1077},[368,33291,33292,33294],{},[371,33293,7347],{},[371,33295,612],{},[738,33297,1092],{"stretchy":1077},[738,33299,1083],{"separator":386},[776,33301],{"width":28059},[368,33303,33304,33306],{},[371,33305,7347],{},[371,33307,612],{},[738,33309,13856],{},[371,33311,1742],{},[738,33313,1078],{"stretchy":1077},[371,33315,7347],{},[738,33317,1092],{"stretchy":1077},[378,33319,33320],{"encoding":380},"\\begin{aligned}\n\\mathbb{E}(f(x))_{x \\sim p(x)} &= \\sum_{x} f(x) \\cdot p(x) \\\\\n&= \\sum_{x} f(x) \\cdot p(x) \\frac{q(x)}{q(x)} \\\\\n&= \\sum_{x} f(x) \\frac{p(x)}{q(x)} \\cdot q(x) \\\\\n&= \\mathbb{E}\\left(f(x) \\frac{p(x)}{q(x)}\\right)_{x \\sim q(x)} \\\\\n&\\approx \\frac{1}{N} \\sum_{n=1}^{N} f(x_n) \\frac{p(x_n)}{q(x_n)}, \\quad x_n \\sim q(x)\n\\end{aligned}",[63,33322,33324],{"className":33323,"ariaHidden":386},[385],[63,33325,33327,33331],{"className":33326},[390],[63,33328],{"className":33329,"style":33330},[394],"height:14.8742em;vertical-align:-7.1871em;",[63,33332,33334],{"className":33333},[399],[63,33335,33337,33484],{"className":33336},[14507],[63,33338,33340],{"className":33339},[15298],[63,33341,33343,33475],{"className":33342},[411,412],[63,33344,33346,33472],{"className":33345},[416],[63,33347,33350,33436,33445,33454,33463],{"className":33348,"style":33349},[420],"height:7.6871em;",[63,33351,33353,33357],{"style":33352},"top:-10.4654em;",[63,33354],{"className":33355,"style":33356},[428],"height:3.8283em;",[63,33358,33360,33363,33366,33369,33372,33375,33378],{"className":33359},[399],[63,33361,5688],{"className":33362},[399,798],[63,33364,1078],{"className":33365},[403],[63,33367,11741],{"className":33368,"style":13736},[399,472],[63,33370,1078],{"className":33371},[403],[63,33373,7347],{"className":33374},[399,472],[63,33376,1092],{"className":33377},[593],[63,33379,33381,33384],{"className":33380},[593],[63,33382,1092],{"className":33383},[593],[63,33385,33387],{"className":33386},[476],[63,33388,33390,33428],{"className":33389},[411,412],[63,33391,33393,33425],{"className":33392},[416],[63,33394,33396],{"className":33395,"style":486},[420],[63,33397,33398,33401],{"style":14284},[63,33399],{"className":33400,"style":817},[428],[63,33402,33404],{"className":33403},[433,434,435,436],[63,33405,33407,33410,33413,33416,33419,33422],{"className":33406},[399,436],[63,33408,7347],{"className":33409},[399,472,436],[63,33411,13856],{"className":33412},[781,436],[63,33414,14],{"className":33415},[399,472,436],[63,33417,1078],{"className":33418},[403,436],[63,33420,7347],{"className":33421},[399,472,436],[63,33423,1092],{"className":33424},[593,436],[63,33426,507],{"className":33427},[506],[63,33429,33431],{"className":33430},[416],[63,33432,33434],{"className":33433,"style":6153},[420],[63,33435],{},[63,33437,33439,33442],{"style":33438},"top:-7.4884em;",[63,33440],{"className":33441,"style":33356},[428],[63,33443],{"className":33444},[399],[63,33446,33448,33451],{"style":33447},"top:-4.5114em;",[63,33449],{"className":33450,"style":33356},[428],[63,33452],{"className":33453},[399],[63,33455,33457,33460],{"style":33456},"top:-1.5114em;",[63,33458],{"className":33459,"style":33356},[428],[63,33461],{"className":33462},[399],[63,33464,33466,33469],{"style":33465},"top:1.7917em;",[63,33467],{"className":33468,"style":33356},[428],[63,33470],{"className":33471},[399],[63,33473,507],{"className":33474},[506],[63,33476,33478],{"className":33477},[416],[63,33479,33482],{"className":33480,"style":33481},[420],"height:7.1871em;",[63,33483],{},[63,33485,33487],{"className":33486},[15522],[63,33488,33490,34587],{"className":33489},[411,412],[63,33491,33493,34584],{"className":33492},[416],[63,33494,33496,33599,33782,33965,34156],{"className":33495,"style":33349},[420],[63,33497,33498,33501],{"style":33352},[63,33499],{"className":33500,"style":33356},[428],[63,33502,33504,33507,33510,33513,33516,33563,33566,33569,33572,33575,33578,33581,33584,33587,33590,33593,33596],{"className":33503},[399],[63,33505],{"className":33506},[399],[63,33508],{"className":33509,"style":777},[776],[63,33511,204],{"className":33512},[781],[63,33514],{"className":33515,"style":777},[776],[63,33517,33519],{"className":33518},[1212,14383],[63,33520,33522,33555],{"className":33521},[411,412],[63,33523,33525,33552],{"className":33524},[416],[63,33526,33528,33542],{"className":33527,"style":15570},[420],[63,33529,33530,33533],{"style":15573},[63,33531],{"className":33532,"style":15577},[428],[63,33534,33536],{"className":33535},[433,434,435,436],[63,33537,33539],{"className":33538},[399,436],[63,33540,7347],{"className":33541},[399,472,436],[63,33543,33544,33547],{"style":15586},[63,33545],{"className":33546,"style":15577},[428],[63,33548,33549],{},[63,33550,1874],{"className":33551},[1212,1958,15595],[63,33553,507],{"className":33554},[506],[63,33556,33558],{"className":33557},[416],[63,33559,33561],{"className":33560,"style":15605},[420],[63,33562],{},[63,33564],{"className":33565,"style":1177},[776],[63,33567,11741],{"className":33568,"style":13736},[399,472],[63,33570,1078],{"className":33571},[403],[63,33573,7347],{"className":33574},[399,472],[63,33576,1092],{"className":33577},[593],[63,33579],{"className":33580,"style":1920},[776],[63,33582,1864],{"className":33583},[830],[63,33585],{"className":33586,"style":1920},[776],[63,33588,14],{"className":33589},[399,472],[63,33591,1078],{"className":33592},[403],[63,33594,7347],{"className":33595},[399,472],[63,33597,1092],{"className":33598},[593],[63,33600,33601,33604],{"style":33438},[63,33602],{"className":33603,"style":33356},[428],[63,33605,33607,33610,33613,33616,33619,33666,33669,33672,33675,33678,33681,33684,33687,33690,33693,33696,33699,33702],{"className":33606},[399],[63,33608],{"className":33609},[399],[63,33611],{"className":33612,"style":777},[776],[63,33614,204],{"className":33615},[781],[63,33617],{"className":33618,"style":777},[776],[63,33620,33622],{"className":33621},[1212,14383],[63,33623,33625,33658],{"className":33624},[411,412],[63,33626,33628,33655],{"className":33627},[416],[63,33629,33631,33645],{"className":33630,"style":15570},[420],[63,33632,33633,33636],{"style":15573},[63,33634],{"className":33635,"style":15577},[428],[63,33637,33639],{"className":33638},[433,434,435,436],[63,33640,33642],{"className":33641},[399,436],[63,33643,7347],{"className":33644},[399,472,436],[63,33646,33647,33650],{"style":15586},[63,33648],{"className":33649,"style":15577},[428],[63,33651,33652],{},[63,33653,1874],{"className":33654},[1212,1958,15595],[63,33656,507],{"className":33657},[506],[63,33659,33661],{"className":33660},[416],[63,33662,33664],{"className":33663,"style":15605},[420],[63,33665],{},[63,33667],{"className":33668,"style":1177},[776],[63,33670,11741],{"className":33671,"style":13736},[399,472],[63,33673,1078],{"className":33674},[403],[63,33676,7347],{"className":33677},[399,472],[63,33679,1092],{"className":33680},[593],[63,33682],{"className":33683,"style":1920},[776],[63,33685,1864],{"className":33686},[830],[63,33688],{"className":33689,"style":1920},[776],[63,33691,14],{"className":33692},[399,472],[63,33694,1078],{"className":33695},[403],[63,33697,7347],{"className":33698},[399,472],[63,33700,1092],{"className":33701},[593],[63,33703,33705,33708,33779],{"className":33704},[399],[63,33706],{"className":33707},[403,404],[63,33709,33711],{"className":33710},[358],[63,33712,33714,33771],{"className":33713},[411,412],[63,33715,33717,33768],{"className":33716},[416],[63,33718,33720,33740,33748],{"className":33719,"style":15961},[420],[63,33721,33722,33725],{"style":15964},[63,33723],{"className":33724,"style":429},[428],[63,33726,33728,33731,33734,33737],{"className":33727},[399],[63,33729,1742],{"className":33730,"style":718},[399,472],[63,33732,1078],{"className":33733},[403],[63,33735,7347],{"className":33736},[399,472],[63,33738,1092],{"className":33739},[593],[63,33741,33742,33745],{"style":554},[63,33743],{"className":33744,"style":429},[428],[63,33746],{"className":33747,"style":562},[561],[63,33749,33750,33753],{"style":16030},[63,33751],{"className":33752,"style":429},[428],[63,33754,33756,33759,33762,33765],{"className":33755},[399],[63,33757,1742],{"className":33758,"style":718},[399,472],[63,33760,1078],{"className":33761},[403],[63,33763,7347],{"className":33764},[399,472],[63,33766,1092],{"className":33767},[593],[63,33769,507],{"className":33770},[506],[63,33772,33774],{"className":33773},[416],[63,33775,33777],{"className":33776,"style":16095},[420],[63,33778],{},[63,33780],{"className":33781},[593,404],[63,33783,33784,33787],{"style":33447},[63,33785],{"className":33786,"style":33356},[428],[63,33788,33790,33793,33796,33799,33802,33849,33852,33855,33858,33861,33864,33944,33947,33950,33953,33956,33959,33962],{"className":33789},[399],[63,33791],{"className":33792},[399],[63,33794],{"className":33795,"style":777},[776],[63,33797,204],{"className":33798},[781],[63,33800],{"className":33801,"style":777},[776],[63,33803,33805],{"className":33804},[1212,14383],[63,33806,33808,33841],{"className":33807},[411,412],[63,33809,33811,33838],{"className":33810},[416],[63,33812,33814,33828],{"className":33813,"style":15570},[420],[63,33815,33816,33819],{"style":15573},[63,33817],{"className":33818,"style":15577},[428],[63,33820,33822],{"className":33821},[433,434,435,436],[63,33823,33825],{"className":33824},[399,436],[63,33826,7347],{"className":33827},[399,472,436],[63,33829,33830,33833],{"style":15586},[63,33831],{"className":33832,"style":15577},[428],[63,33834,33835],{},[63,33836,1874],{"className":33837},[1212,1958,15595],[63,33839,507],{"className":33840},[506],[63,33842,33844],{"className":33843},[416],[63,33845,33847],{"className":33846,"style":15605},[420],[63,33848],{},[63,33850],{"className":33851,"style":1177},[776],[63,33853,11741],{"className":33854,"style":13736},[399,472],[63,33856,1078],{"className":33857},[403],[63,33859,7347],{"className":33860},[399,472],[63,33862,1092],{"className":33863},[593],[63,33865,33867,33870,33941],{"className":33866},[399],[63,33868],{"className":33869},[403,404],[63,33871,33873],{"className":33872},[358],[63,33874,33876,33933],{"className":33875},[411,412],[63,33877,33879,33930],{"className":33878},[416],[63,33880,33882,33902,33910],{"className":33881,"style":15961},[420],[63,33883,33884,33887],{"style":15964},[63,33885],{"className":33886,"style":429},[428],[63,33888,33890,33893,33896,33899],{"className":33889},[399],[63,33891,1742],{"className":33892,"style":718},[399,472],[63,33894,1078],{"className":33895},[403],[63,33897,7347],{"className":33898},[399,472],[63,33900,1092],{"className":33901},[593],[63,33903,33904,33907],{"style":554},[63,33905],{"className":33906,"style":429},[428],[63,33908],{"className":33909,"style":562},[561],[63,33911,33912,33915],{"style":16030},[63,33913],{"className":33914,"style":429},[428],[63,33916,33918,33921,33924,33927],{"className":33917},[399],[63,33919,14],{"className":33920},[399,472],[63,33922,1078],{"className":33923},[403],[63,33925,7347],{"className":33926},[399,472],[63,33928,1092],{"className":33929},[593],[63,33931,507],{"className":33932},[506],[63,33934,33936],{"className":33935},[416],[63,33937,33939],{"className":33938,"style":16095},[420],[63,33940],{},[63,33942],{"className":33943},[593,404],[63,33945],{"className":33946,"style":1920},[776],[63,33948,1864],{"className":33949},[830],[63,33951],{"className":33952,"style":1920},[776],[63,33954,1742],{"className":33955,"style":718},[399,472],[63,33957,1078],{"className":33958},[403],[63,33960,7347],{"className":33961},[399,472],[63,33963,1092],{"className":33964},[593],[63,33966,33967,33970],{"style":33456},[63,33968],{"className":33969,"style":33356},[428],[63,33971,33973,33976,33979,33982,33985,33988,33991],{"className":33972},[399],[63,33974],{"className":33975},[399],[63,33977],{"className":33978,"style":777},[776],[63,33980,204],{"className":33981},[781],[63,33983],{"className":33984,"style":777},[776],[63,33986,5688],{"className":33987},[399,798],[63,33989],{"className":33990,"style":1177},[776],[63,33992,33994,34101],{"className":33993},[1222],[63,33995,33997,34003,34006,34009,34012,34015,34095],{"className":33996},[1222],[63,33998,34000],{"className":33999,"style":1227},[403,1226],[63,34001,1078],{"className":34002},[1231,435],[63,34004,11741],{"className":34005,"style":13736},[399,472],[63,34007,1078],{"className":34008},[403],[63,34010,7347],{"className":34011},[399,472],[63,34013,1092],{"className":34014},[593],[63,34016,34018,34021,34092],{"className":34017},[399],[63,34019],{"className":34020},[403,404],[63,34022,34024],{"className":34023},[358],[63,34025,34027,34084],{"className":34026},[411,412],[63,34028,34030,34081],{"className":34029},[416],[63,34031,34033,34053,34061],{"className":34032,"style":15961},[420],[63,34034,34035,34038],{"style":15964},[63,34036],{"className":34037,"style":429},[428],[63,34039,34041,34044,34047,34050],{"className":34040},[399],[63,34042,1742],{"className":34043,"style":718},[399,472],[63,34045,1078],{"className":34046},[403],[63,34048,7347],{"className":34049},[399,472],[63,34051,1092],{"className":34052},[593],[63,34054,34055,34058],{"style":554},[63,34056],{"className":34057,"style":429},[428],[63,34059],{"className":34060,"style":562},[561],[63,34062,34063,34066],{"style":16030},[63,34064],{"className":34065,"style":429},[428],[63,34067,34069,34072,34075,34078],{"className":34068},[399],[63,34070,14],{"className":34071},[399,472],[63,34073,1078],{"className":34074},[403],[63,34076,7347],{"className":34077},[399,472],[63,34079,1092],{"className":34080},[593],[63,34082,507],{"className":34083},[506],[63,34085,34087],{"className":34086},[416],[63,34088,34090],{"className":34089,"style":16095},[420],[63,34091],{},[63,34093],{"className":34094},[593,404],[63,34096,34098],{"className":34097,"style":1227},[593,1226],[63,34099,1092],{"className":34100},[1231,435],[63,34102,34104],{"className":34103},[476],[63,34105,34107,34147],{"className":34106},[411,412],[63,34108,34110,34144],{"className":34109},[416],[63,34111,34114],{"className":34112,"style":34113},[420],"height:-0.4747em;",[63,34115,34117,34120],{"style":34116},"top:-1.7003em;margin-right:0.05em;",[63,34118],{"className":34119,"style":817},[428],[63,34121,34123],{"className":34122},[433,434,435,436],[63,34124,34126,34129,34132,34135,34138,34141],{"className":34125},[399,436],[63,34127,7347],{"className":34128},[399,472,436],[63,34130,13856],{"className":34131},[781,436],[63,34133,1742],{"className":34134,"style":718},[399,472,436],[63,34136,1078],{"className":34137},[403,436],[63,34139,7347],{"className":34140},[399,472,436],[63,34142,1092],{"className":34143},[593,436],[63,34145,507],{"className":34146},[506],[63,34148,34150],{"className":34149},[416],[63,34151,34154],{"className":34152,"style":34153},[420],"height:1.1747em;",[63,34155],{},[63,34157,34158,34161],{"style":33465},[63,34159],{"className":34160,"style":33356},[428],[63,34162,34164,34167,34170,34173,34176,34238,34241,34308,34311,34314,34317,34357,34360,34514,34517,34520,34523,34563,34566,34569,34572,34575,34578,34581],{"className":34163},[399],[63,34165],{"className":34166},[399],[63,34168],{"className":34169,"style":777},[776],[63,34171,14820],{"className":34172},[781],[63,34174],{"className":34175,"style":777},[776],[63,34177,34179,34182,34235],{"className":34178},[399],[63,34180],{"className":34181},[403,404],[63,34183,34185],{"className":34184},[358],[63,34186,34188,34227],{"className":34187},[411,412],[63,34189,34191,34224],{"className":34190},[416],[63,34192,34194,34205,34213],{"className":34193,"style":16424},[420],[63,34195,34196,34199],{"style":15964},[63,34197],{"className":34198,"style":429},[428],[63,34200,34202],{"className":34201},[399],[63,34203,7342],{"className":34204,"style":16436},[399,472],[63,34206,34207,34210],{"style":554},[63,34208],{"className":34209,"style":429},[428],[63,34211],{"className":34212,"style":562},[561],[63,34214,34215,34218],{"style":16030},[63,34216],{"className":34217,"style":429},[428],[63,34219,34221],{"className":34220},[399],[63,34222,363],{"className":34223},[399],[63,34225,507],{"className":34226},[506],[63,34228,34230],{"className":34229},[416],[63,34231,34233],{"className":34232,"style":16465},[420],[63,34234],{},[63,34236],{"className":34237},[593,404],[63,34239],{"className":34240,"style":1177},[776],[63,34242,34244],{"className":34243},[1212,14383],[63,34245,34247,34300],{"className":34246},[411,412],[63,34248,34250,34297],{"className":34249},[416],[63,34251,34253,34273,34283],{"className":34252,"style":16486},[420],[63,34254,34255,34258],{"style":16489},[63,34256],{"className":34257,"style":15577},[428],[63,34259,34261],{"className":34260},[433,434,435,436],[63,34262,34264,34267,34270],{"className":34263},[399,436],[63,34265,612],{"className":34266},[399,472,436],[63,34268,204],{"className":34269},[781,436],[63,34271,363],{"className":34272},[399,436],[63,34274,34275,34278],{"style":15586},[63,34276],{"className":34277,"style":15577},[428],[63,34279,34280],{},[63,34281,1874],{"className":34282},[1212,1958,15595],[63,34284,34285,34288],{"style":16520},[63,34286],{"className":34287,"style":15577},[428],[63,34289,34291],{"className":34290},[433,434,435,436],[63,34292,34294],{"className":34293},[399,436],[63,34295,7342],{"className":34296,"style":16436},[399,472,436],[63,34298,507],{"className":34299},[506],[63,34301,34303],{"className":34302},[416],[63,34304,34306],{"className":34305,"style":16539},[420],[63,34307],{},[63,34309],{"className":34310,"style":1177},[776],[63,34312,11741],{"className":34313,"style":13736},[399,472],[63,34315,1078],{"className":34316},[403],[63,34318,34320,34323],{"className":34319},[399],[63,34321,7347],{"className":34322},[399,472],[63,34324,34326],{"className":34325},[476],[63,34327,34329,34349],{"className":34328},[411,412],[63,34330,34332,34346],{"className":34331},[416],[63,34333,34335],{"className":34334,"style":8342},[420],[63,34336,34337,34340],{"style":1535},[63,34338],{"className":34339,"style":817},[428],[63,34341,34343],{"className":34342},[433,434,435,436],[63,34344,612],{"className":34345},[399,472,436],[63,34347,507],{"className":34348},[506],[63,34350,34352],{"className":34351},[416],[63,34353,34355],{"className":34354,"style":1554},[420],[63,34356],{},[63,34358,1092],{"className":34359},[593],[63,34361,34363,34366,34511],{"className":34362},[399],[63,34364],{"className":34365},[403,404],[63,34367,34369],{"className":34368},[358],[63,34370,34372,34503],{"className":34371},[411,412],[63,34373,34375,34500],{"className":34374},[416],[63,34376,34378,34435,34443],{"className":34377,"style":15961},[420],[63,34379,34380,34383],{"style":15964},[63,34381],{"className":34382,"style":429},[428],[63,34384,34386,34389,34392,34432],{"className":34385},[399],[63,34387,1742],{"className":34388,"style":718},[399,472],[63,34390,1078],{"className":34391},[403],[63,34393,34395,34398],{"className":34394},[399],[63,34396,7347],{"className":34397},[399,472],[63,34399,34401],{"className":34400},[476],[63,34402,34404,34424],{"className":34403},[411,412],[63,34405,34407,34421],{"className":34406},[416],[63,34408,34410],{"className":34409,"style":8342},[420],[63,34411,34412,34415],{"style":1535},[63,34413],{"className":34414,"style":817},[428],[63,34416,34418],{"className":34417},[433,434,435,436],[63,34419,612],{"className":34420},[399,472,436],[63,34422,507],{"className":34423},[506],[63,34425,34427],{"className":34426},[416],[63,34428,34430],{"className":34429,"style":1554},[420],[63,34431],{},[63,34433,1092],{"className":34434},[593],[63,34436,34437,34440],{"style":554},[63,34438],{"className":34439,"style":429},[428],[63,34441],{"className":34442,"style":562},[561],[63,34444,34445,34448],{"style":16030},[63,34446],{"className":34447,"style":429},[428],[63,34449,34451,34454,34457,34497],{"className":34450},[399],[63,34452,14],{"className":34453},[399,472],[63,34455,1078],{"className":34456},[403],[63,34458,34460,34463],{"className":34459},[399],[63,34461,7347],{"className":34462},[399,472],[63,34464,34466],{"className":34465},[476],[63,34467,34469,34489],{"className":34468},[411,412],[63,34470,34472,34486],{"className":34471},[416],[63,34473,34475],{"className":34474,"style":8342},[420],[63,34476,34477,34480],{"style":1535},[63,34478],{"className":34479,"style":817},[428],[63,34481,34483],{"className":34482},[433,434,435,436],[63,34484,612],{"className":34485},[399,472,436],[63,34487,507],{"className":34488},[506],[63,34490,34492],{"className":34491},[416],[63,34493,34495],{"className":34494,"style":1554},[420],[63,34496],{},[63,34498,1092],{"className":34499},[593],[63,34501,507],{"className":34502},[506],[63,34504,34506],{"className":34505},[416],[63,34507,34509],{"className":34508,"style":16095},[420],[63,34510],{},[63,34512],{"className":34513},[593,404],[63,34515,1083],{"className":34516},[1173],[63,34518],{"className":34519,"style":29024},[776],[63,34521],{"className":34522,"style":1177},[776],[63,34524,34526,34529],{"className":34525},[399],[63,34527,7347],{"className":34528},[399,472],[63,34530,34532],{"className":34531},[476],[63,34533,34535,34555],{"className":34534},[411,412],[63,34536,34538,34552],{"className":34537},[416],[63,34539,34541],{"className":34540,"style":8342},[420],[63,34542,34543,34546],{"style":1535},[63,34544],{"className":34545,"style":817},[428],[63,34547,34549],{"className":34548},[433,434,435,436],[63,34550,612],{"className":34551},[399,472,436],[63,34553,507],{"className":34554},[506],[63,34556,34558],{"className":34557},[416],[63,34559,34561],{"className":34560,"style":1554},[420],[63,34562],{},[63,34564],{"className":34565,"style":777},[776],[63,34567,13856],{"className":34568},[781],[63,34570],{"className":34571,"style":777},[776],[63,34573,1742],{"className":34574,"style":718},[399,472],[63,34576,1078],{"className":34577},[403],[63,34579,7347],{"className":34580},[399,472],[63,34582,1092],{"className":34583},[593],[63,34585,507],{"className":34586},[506],[63,34588,34590],{"className":34589},[416],[63,34591,34593],{"className":34592,"style":33481},[420],[63,34594],{},[14,34596,34597],{},"利用重要性采样，我们就可以把 on-policy 的梯度公式改写为可以复用旧数据的 off-policy 形式。",[30,34599,34600],{"id":32585},"Off-policy",[14,34602,34603,34604,34667,34668,147,34671,34667,34700,34703,34704,34841,34842,35033,35034,20807,35037,35252],{},"记 ",[63,34605,34607,34625],{"className":34606},[342],[63,34608,34610],{"className":34609},[346],[348,34611,34612],{"xmlns":350},[352,34613,34614,34622],{},[355,34615,34616],{},[742,34617,34618,34620],{},[371,34619,14101],{},[738,34621,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[378,34623,34624],{"encoding":380},"\\theta'",[63,34626,34628],{"className":34627,"ariaHidden":386},[385],[63,34629,34631,34635],{"className":34630},[390],[63,34632],{"className":34633,"style":34634},[394],"height:0.7519em;",[63,34636,34638,34641],{"className":34637},[399],[63,34639,14101],{"className":34640,"style":3276},[399,472],[63,34642,34644],{"className":34643},[476],[63,34645,34647],{"className":34646},[411],[63,34648,34650],{"className":34649},[416],[63,34651,34653],{"className":34652,"style":34634},[420],[63,34654,34655,34658],{"style":813},[63,34656],{"className":34657,"style":817},[428],[63,34659,34661],{"className":34660},[433,434,435,436],[63,34662,34664],{"className":34663},[399,436],[63,34665,19131],{"className":34666},[399,436]," 为 ",[11217,34669,34670],{},"采集数据时使用的旧策略",[63,34672,34674,34688],{"className":34673},[342],[63,34675,34677],{"className":34676},[346],[348,34678,34679],{"xmlns":350},[352,34680,34681,34685],{},[355,34682,34683],{},[371,34684,14101],{},[378,34686,34687],{"encoding":380},"\\theta",[63,34689,34691],{"className":34690,"ariaHidden":386},[385],[63,34692,34694,34697],{"className":34693},[390],[63,34695],{"className":34696,"style":684},[394],[63,34698,14101],{"className":34699,"style":3276},[399,472],[11217,34701,34702],{},"当前要优化的策略","；优势 ",[63,34705,34707,34737],{"className":34706},[342],[63,34708,34710],{"className":34709},[346],[348,34711,34712],{"xmlns":350},[352,34713,34714,34734],{},[355,34715,34716],{},[1870,34717,34718,34720,34726],{},[371,34719,1054],{},[742,34721,34722,34724],{},[371,34723,14101],{},[738,34725,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,34727,34728,34730,34732],{},[371,34729,27970],{},[371,34731,1054],{},[371,34733,5688],{},[378,34735,34736],{"encoding":380},"A_{\\theta'}^{GAE}",[63,34738,34740],{"className":34739,"ariaHidden":386},[385],[63,34741,34743,34747],{"className":34742},[390],[63,34744],{"className":34745,"style":34746},[394],"height:1.1244em;vertical-align:-0.2831em;",[63,34748,34750,34753],{"className":34749},[399],[63,34751,1054],{"className":34752},[399,472],[63,34754,34756],{"className":34755},[476],[63,34757,34759,34832],{"className":34758},[411,412],[63,34760,34762,34829],{"className":34761},[416],[63,34763,34765,34809],{"className":34764,"style":3264},[420],[63,34766,34768,34771],{"style":34767},"top:-2.4169em;margin-left:0em;margin-right:0.05em;",[63,34769],{"className":34770,"style":817},[428],[63,34772,34774],{"className":34773},[433,434,435,436],[63,34775,34777],{"className":34776},[399,436],[63,34778,34780,34783],{"className":34779},[399,436],[63,34781,14101],{"className":34782,"style":3276},[399,472,436],[63,34784,34786],{"className":34785},[476],[63,34787,34789],{"className":34788},[411],[63,34790,34792],{"className":34791},[416],[63,34793,34795],{"className":34794,"style":19291},[420],[63,34796,34797,34800],{"style":19294},[63,34798],{"className":34799,"style":493},[428],[63,34801,34803],{"className":34802},[433,497,498,436],[63,34804,34806],{"className":34805},[399,436],[63,34807,19131],{"className":34808},[399,436],[63,34810,34811,34814],{"style":813},[63,34812],{"className":34813,"style":817},[428],[63,34815,34817],{"className":34816},[433,434,435,436],[63,34818,34820,34823,34826],{"className":34819},[399,436],[63,34821,27970],{"className":34822},[399,472,436],[63,34824,1054],{"className":34825},[399,472,436],[63,34827,5688],{"className":34828,"style":6101},[399,472,436],[63,34830,507],{"className":34831},[506],[63,34833,34835],{"className":34834},[416],[63,34836,34839],{"className":34837,"style":34838},[420],"height:0.2831em;",[63,34840],{}," 由旧策略下的价值网络估计而来。结合恒等式 ",[63,34843,34845,34899],{"className":34844},[342],[63,34846,34848],{"className":34847},[346],[348,34849,34850],{"xmlns":350},[352,34851,34852,34896],{},[355,34853,34854,34856,34858,34860,34862,34864,34866,34868,34870],{},[371,34855,14525],{"mathvariant":1053},[371,34857,14950],{},[738,34859,1100],{},[371,34861,11741],{},[738,34863,1078],{"stretchy":1077},[371,34865,7347],{},[738,34867,1092],{"stretchy":1077},[738,34869,204],{},[14519,34871,34872],{"scriptlevel":1802,"displaystyle":386},[358,34873,34874,34886],{},[355,34875,34876,34878,34880,34882,34884],{},[371,34877,14525],{"mathvariant":1053},[371,34879,11741],{},[738,34881,1078],{"stretchy":1077},[371,34883,7347],{},[738,34885,1092],{"stretchy":1077},[355,34887,34888,34890,34892,34894],{},[371,34889,11741],{},[738,34891,1078],{"stretchy":1077},[371,34893,7347],{},[738,34895,1092],{"stretchy":1077},[378,34897,34898],{"encoding":380},"\\nabla \\log f(x) = \\dfrac{\\nabla f(x)}{f(x)}",[63,34900,34902,34943],{"className":34901,"ariaHidden":386},[385],[63,34903,34905,34908,34911,34914,34919,34922,34925,34928,34931,34934,34937,34940],{"className":34904},[390],[63,34906],{"className":34907,"style":1155},[394],[63,34909,14525],{"className":34910},[399],[63,34912],{"className":34913,"style":1177},[776],[63,34915,16996,34917],{"className":34916},[1212],[63,34918,17000],{"style":16999},[63,34920],{"className":34921,"style":1177},[776],[63,34923,11741],{"className":34924,"style":13736},[399,472],[63,34926,1078],{"className":34927},[403],[63,34929,7347],{"className":34930},[399,472],[63,34932,1092],{"className":34933},[593],[63,34935],{"className":34936,"style":777},[776],[63,34938,204],{"className":34939},[781],[63,34941],{"className":34942,"style":777},[776],[63,34944,34946,34950],{"className":34945},[390],[63,34947],{"className":34948,"style":34949},[394],"height:2.363em;vertical-align:-0.936em;",[63,34951,34953,34956,35030],{"className":34952},[399],[63,34954],{"className":34955},[403,404],[63,34957,34959],{"className":34958},[358],[63,34960,34962,35022],{"className":34961},[411,412],[63,34963,34965,35019],{"className":34964},[416],[63,34966,34968,34988,34996],{"className":34967,"style":15961},[420],[63,34969,34970,34973],{"style":15964},[63,34971],{"className":34972,"style":429},[428],[63,34974,34976,34979,34982,34985],{"className":34975},[399],[63,34977,11741],{"className":34978,"style":13736},[399,472],[63,34980,1078],{"className":34981},[403],[63,34983,7347],{"className":34984},[399,472],[63,34986,1092],{"className":34987},[593],[63,34989,34990,34993],{"style":554},[63,34991],{"className":34992,"style":429},[428],[63,34994],{"className":34995,"style":562},[561],[63,34997,34998,35001],{"style":16030},[63,34999],{"className":35000,"style":429},[428],[63,35002,35004,35007,35010,35013,35016],{"className":35003},[399],[63,35005,14525],{"className":35006},[399],[63,35008,11741],{"className":35009,"style":13736},[399,472],[63,35011,1078],{"className":35012},[403],[63,35014,7347],{"className":35015},[399,472],[63,35017,1092],{"className":35018},[593],[63,35020,507],{"className":35021},[506],[63,35023,35025],{"className":35024},[416],[63,35026,35028],{"className":35027,"style":16095},[420],[63,35029],{},[63,35031],{"className":35032},[593,404],"，可以把策略梯度写成包含 ",[11217,35035,35036],{},"重要性采样比",[63,35038,35040,35072],{"className":35039},[342],[63,35041,35043],{"className":35042},[346],[348,35044,35045],{"xmlns":350},[352,35046,35047,35069],{},[355,35048,35049],{},[14519,35050,35051],{"scriptlevel":1802,"displaystyle":386},[358,35052,35053,35059],{},[368,35054,35055,35057],{},[371,35056,5685],{},[371,35058,14101],{},[368,35060,35061,35063],{},[371,35062,5685],{},[742,35064,35065,35067],{},[371,35066,14101],{},[738,35068,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[378,35070,35071],{"encoding":380},"\\dfrac{P_\\theta}{P_{\\theta'}}",[63,35073,35075],{"className":35074,"ariaHidden":386},[385],[63,35076,35078,35082],{"className":35077},[390],[63,35079],{"className":35080,"style":35081},[394],"height:2.1963em;vertical-align:-0.836em;",[63,35083,35085,35088,35249],{"className":35084},[399],[63,35086],{"className":35087},[403,404],[63,35089,35091],{"className":35090},[358],[63,35092,35094,35240],{"className":35093},[411,412],[63,35095,35097,35237],{"className":35096},[416],[63,35098,35101,35181,35189],{"className":35099,"style":35100},[420],"height:1.3603em;",[63,35102,35103,35106],{"style":15964},[63,35104],{"className":35105,"style":429},[428],[63,35107,35109],{"className":35108},[399],[63,35110,35112,35115],{"className":35111},[399],[63,35113,5685],{"className":35114,"style":3251},[399,472],[63,35116,35118],{"className":35117},[476],[63,35119,35121,35173],{"className":35120},[411,412],[63,35122,35124,35170],{"className":35123},[416],[63,35125,35127],{"className":35126,"style":1532},[420],[63,35128,35129,35132],{"style":7483},[63,35130],{"className":35131,"style":817},[428],[63,35133,35135],{"className":35134},[433,434,435,436],[63,35136,35138],{"className":35137},[399,436],[63,35139,35141,35144],{"className":35140},[399,436],[63,35142,14101],{"className":35143,"style":3276},[399,472,436],[63,35145,35147],{"className":35146},[476],[63,35148,35150],{"className":35149},[411],[63,35151,35153],{"className":35152},[416],[63,35154,35156],{"className":35155,"style":19291},[420],[63,35157,35158,35161],{"style":19294},[63,35159],{"className":35160,"style":493},[428],[63,35162,35164],{"className":35163},[433,497,498,436],[63,35165,35167],{"className":35166},[399,436],[63,35168,19131],{"className":35169},[399,436],[63,35171,507],{"className":35172},[506],[63,35174,35176],{"className":35175},[416],[63,35177,35179],{"className":35178,"style":1554},[420],[63,35180],{},[63,35182,35183,35186],{"style":554},[63,35184],{"className":35185,"style":429},[428],[63,35187],{"className":35188,"style":562},[561],[63,35190,35191,35194],{"style":16030},[63,35192],{"className":35193,"style":429},[428],[63,35195,35197],{"className":35196},[399],[63,35198,35200,35203],{"className":35199},[399],[63,35201,5685],{"className":35202,"style":3251},[399,472],[63,35204,35206],{"className":35205},[476],[63,35207,35209,35229],{"className":35208},[411,412],[63,35210,35212,35226],{"className":35211},[416],[63,35213,35215],{"className":35214,"style":1532},[420],[63,35216,35217,35220],{"style":7483},[63,35218],{"className":35219,"style":817},[428],[63,35221,35223],{"className":35222},[433,434,435,436],[63,35224,14101],{"className":35225,"style":3276},[399,472,436],[63,35227,507],{"className":35228},[506],[63,35230,35232],{"className":35231},[416],[63,35233,35235],{"className":35234,"style":1554},[420],[63,35236],{},[63,35238,507],{"className":35239},[506],[63,35241,35243],{"className":35242},[416],[63,35244,35247],{"className":35245,"style":35246},[420],"height:0.836em;",[63,35248],{},[63,35250],{"className":35251},[593,404]," 的形式：",[63,35254,35256],{"className":35255},[14495],[63,35257,35259,35311],{"className":35258},[342],[63,35260,35262],{"className":35261},[346],[348,35263,35264],{"xmlns":350,"display":9561},[352,35265,35266,35308],{},[355,35267,35268,35270,35272,35274,35276,35278,35280,35282,35284],{},[371,35269,14525],{"mathvariant":1053},[371,35271,14950],{},[738,35273,1100],{},[371,35275,11741],{},[738,35277,1078],{"stretchy":1077},[371,35279,7347],{},[738,35281,1092],{"stretchy":1077},[738,35283,204],{},[358,35285,35286,35298],{},[355,35287,35288,35290,35292,35294,35296],{},[371,35289,14525],{"mathvariant":1053},[371,35291,11741],{},[738,35293,1078],{"stretchy":1077},[371,35295,7347],{},[738,35297,1092],{"stretchy":1077},[355,35299,35300,35302,35304,35306],{},[371,35301,11741],{},[738,35303,1078],{"stretchy":1077},[371,35305,7347],{},[738,35307,1092],{"stretchy":1077},[378,35309,35310],{"encoding":380},"\\nabla \\log f(x) = \\frac{\\nabla f(x)}{f(x)}",[63,35312,35314,35355],{"className":35313,"ariaHidden":386},[385],[63,35315,35317,35320,35323,35326,35331,35334,35337,35340,35343,35346,35349,35352],{"className":35316},[390],[63,35318],{"className":35319,"style":1155},[394],[63,35321,14525],{"className":35322},[399],[63,35324],{"className":35325,"style":1177},[776],[63,35327,16996,35329],{"className":35328},[1212],[63,35330,17000],{"style":16999},[63,35332],{"className":35333,"style":1177},[776],[63,35335,11741],{"className":35336,"style":13736},[399,472],[63,35338,1078],{"className":35339},[403],[63,35341,7347],{"className":35342},[399,472],[63,35344,1092],{"className":35345},[593],[63,35347],{"className":35348,"style":777},[776],[63,35350,204],{"className":35351},[781],[63,35353],{"className":35354,"style":777},[776],[63,35356,35358,35361],{"className":35357},[390],[63,35359],{"className":35360,"style":34949},[394],[63,35362,35364,35367,35441],{"className":35363},[399],[63,35365],{"className":35366},[403,404],[63,35368,35370],{"className":35369},[358],[63,35371,35373,35433],{"className":35372},[411,412],[63,35374,35376,35430],{"className":35375},[416],[63,35377,35379,35399,35407],{"className":35378,"style":15961},[420],[63,35380,35381,35384],{"style":15964},[63,35382],{"className":35383,"style":429},[428],[63,35385,35387,35390,35393,35396],{"className":35386},[399],[63,35388,11741],{"className":35389,"style":13736},[399,472],[63,35391,1078],{"className":35392},[403],[63,35394,7347],{"className":35395},[399,472],[63,35397,1092],{"className":35398},[593],[63,35400,35401,35404],{"style":554},[63,35402],{"className":35403,"style":429},[428],[63,35405],{"className":35406,"style":562},[561],[63,35408,35409,35412],{"style":16030},[63,35410],{"className":35411,"style":429},[428],[63,35413,35415,35418,35421,35424,35427],{"className":35414},[399],[63,35416,14525],{"className":35417},[399],[63,35419,11741],{"className":35420,"style":13736},[399,472],[63,35422,1078],{"className":35423},[403],[63,35425,7347],{"className":35426},[399,472],[63,35428,1092],{"className":35429},[593],[63,35431,507],{"className":35432},[506],[63,35434,35436],{"className":35435},[416],[63,35437,35439],{"className":35438,"style":16095},[420],[63,35440],{},[63,35442],{"className":35443},[593,404],[63,35445,35447],{"className":35446},[14495],[63,35448,35450,36172],{"className":35449},[342],[63,35451,35453],{"className":35452},[346],[348,35454,35455],{"xmlns":350,"display":9561},[352,35456,35457,36169],{},[14507,35458,35459,35583,35779,36005],{"rowspacing":14509,"columnalign":14510,"columnspacing":14511},[14513,35460,35461,35467],{},[14516,35462,35463],{},[14519,35464,35465],{"scriptlevel":1802,"displaystyle":386},[355,35466],{},[14516,35468,35469],{},[14519,35470,35471],{"scriptlevel":1802,"displaystyle":386},[355,35472,35473,35475,35481,35495,35513,35527,35529,35537,35539,35547,35549,35551,35553,35555,35561,35563,35571,35573,35581],{},[355,35474],{},[358,35476,35477,35479],{},[361,35478,363],{},[371,35480,7342],{},[14828,35482,35483,35485,35493],{},[738,35484,1874],{},[355,35486,35487,35489,35491],{},[371,35488,612],{},[738,35490,204],{},[361,35492,363],{},[371,35494,7342],{},[14828,35496,35497,35499,35507],{},[738,35498,1874],{},[355,35500,35501,35503,35505],{},[371,35502,1057],{},[738,35504,204],{},[361,35506,363],{},[368,35508,35509,35511],{},[371,35510,13358],{},[371,35512,612],{},[1870,35514,35515,35517,35519],{},[371,35516,1054],{},[371,35518,14101],{},[355,35520,35521,35523,35525],{},[371,35522,27970],{},[371,35524,1054],{},[371,35526,5688],{},[738,35528,1078],{"stretchy":1077},[1870,35530,35531,35533,35535],{},[371,35532,12890],{},[371,35534,612],{},[371,35536,1057],{},[738,35538,1083],{"separator":386},[1870,35540,35541,35543,35545],{},[371,35542,38],{},[371,35544,612],{},[371,35546,1057],{},[738,35548,1092],{"stretchy":1077},[371,35550,14525],{"mathvariant":1053},[371,35552,14950],{},[738,35554,1100],{},[368,35556,35557,35559],{},[371,35558,5685],{},[371,35560,14101],{},[738,35562,1078],{"stretchy":1077},[1870,35564,35565,35567,35569],{},[371,35566,38],{},[371,35568,612],{},[371,35570,1057],{},[371,35572,13030],{"mathvariant":1053},[1870,35574,35575,35577,35579],{},[371,35576,12890],{},[371,35578,612],{},[371,35580,1057],{},[738,35582,1092],{"stretchy":1077},[14513,35584,35585,35591],{},[14516,35586,35587],{},[14519,35588,35589],{"scriptlevel":1802,"displaystyle":386},[355,35590],{},[14516,35592,35593],{},[14519,35594,35595],{"scriptlevel":1802,"displaystyle":386},[355,35596,35597,35599,35601,35607,35621,35639,35657,35659,35667,35669,35677,35679,35745,35747,35749,35751,35757,35759,35767,35769,35777],{},[355,35598],{},[738,35600,204],{},[358,35602,35603,35605],{},[361,35604,363],{},[371,35606,7342],{},[14828,35608,35609,35611,35619],{},[738,35610,1874],{},[355,35612,35613,35615,35617],{},[371,35614,612],{},[738,35616,204],{},[361,35618,363],{},[371,35620,7342],{},[14828,35622,35623,35625,35633],{},[738,35624,1874],{},[355,35626,35627,35629,35631],{},[371,35628,1057],{},[738,35630,204],{},[361,35632,363],{},[368,35634,35635,35637],{},[371,35636,13358],{},[371,35638,612],{},[1870,35640,35641,35643,35649],{},[371,35642,1054],{},[742,35644,35645,35647],{},[371,35646,14101],{},[738,35648,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,35650,35651,35653,35655],{},[371,35652,27970],{},[371,35654,1054],{},[371,35656,5688],{},[738,35658,1078],{"stretchy":1077},[1870,35660,35661,35663,35665],{},[371,35662,12890],{},[371,35664,612],{},[371,35666,1057],{},[738,35668,1083],{"separator":386},[1870,35670,35671,35673,35675],{},[371,35672,38],{},[371,35674,612],{},[371,35676,1057],{},[738,35678,1092],{"stretchy":1077},[358,35680,35681,35711],{},[355,35682,35683,35689,35691,35699,35701,35709],{},[368,35684,35685,35687],{},[371,35686,5685],{},[371,35688,14101],{},[738,35690,1078],{"stretchy":1077},[1870,35692,35693,35695,35697],{},[371,35694,38],{},[371,35696,612],{},[371,35698,1057],{},[371,35700,13030],{"mathvariant":1053},[1870,35702,35703,35705,35707],{},[371,35704,12890],{},[371,35706,612],{},[371,35708,1057],{},[738,35710,1092],{"stretchy":1077},[355,35712,35713,35723,35725,35733,35735,35743],{},[368,35714,35715,35717],{},[371,35716,5685],{},[742,35718,35719,35721],{},[371,35720,14101],{},[738,35722,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,35724,1078],{"stretchy":1077},[1870,35726,35727,35729,35731],{},[371,35728,38],{},[371,35730,612],{},[371,35732,1057],{},[371,35734,13030],{"mathvariant":1053},[1870,35736,35737,35739,35741],{},[371,35738,12890],{},[371,35740,612],{},[371,35742,1057],{},[738,35744,1092],{"stretchy":1077},[371,35746,14525],{"mathvariant":1053},[371,35748,14950],{},[738,35750,1100],{},[368,35752,35753,35755],{},[371,35754,5685],{},[371,35756,14101],{},[738,35758,1078],{"stretchy":1077},[1870,35760,35761,35763,35765],{},[371,35762,38],{},[371,35764,612],{},[371,35766,1057],{},[371,35768,13030],{"mathvariant":1053},[1870,35770,35771,35773,35775],{},[371,35772,12890],{},[371,35774,612],{},[371,35776,1057],{},[738,35778,1092],{"stretchy":1077},[14513,35780,35781,35787],{},[14516,35782,35783],{},[14519,35784,35785],{"scriptlevel":1802,"displaystyle":386},[355,35786],{},[14516,35788,35789],{},[14519,35790,35791],{"scriptlevel":1802,"displaystyle":386},[355,35792,35793,35795,35797,35803,35817,35835,35853,35855,35863,35865,35873,35875,35941],{},[355,35794],{},[738,35796,204],{},[358,35798,35799,35801],{},[361,35800,363],{},[371,35802,7342],{},[14828,35804,35805,35807,35815],{},[738,35806,1874],{},[355,35808,35809,35811,35813],{},[371,35810,612],{},[738,35812,204],{},[361,35814,363],{},[371,35816,7342],{},[14828,35818,35819,35821,35829],{},[738,35820,1874],{},[355,35822,35823,35825,35827],{},[371,35824,1057],{},[738,35826,204],{},[361,35828,363],{},[368,35830,35831,35833],{},[371,35832,13358],{},[371,35834,612],{},[1870,35836,35837,35839,35845],{},[371,35838,1054],{},[742,35840,35841,35843],{},[371,35842,14101],{},[738,35844,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,35846,35847,35849,35851],{},[371,35848,27970],{},[371,35850,1054],{},[371,35852,5688],{},[738,35854,1078],{"stretchy":1077},[1870,35856,35857,35859,35861],{},[371,35858,12890],{},[371,35860,612],{},[371,35862,1057],{},[738,35864,1083],{"separator":386},[1870,35866,35867,35869,35871],{},[371,35868,38],{},[371,35870,612],{},[371,35872,1057],{},[738,35874,1092],{"stretchy":1077},[358,35876,35877,35907],{},[355,35878,35879,35885,35887,35895,35897,35905],{},[368,35880,35881,35883],{},[371,35882,5685],{},[371,35884,14101],{},[738,35886,1078],{"stretchy":1077},[1870,35888,35889,35891,35893],{},[371,35890,38],{},[371,35892,612],{},[371,35894,1057],{},[371,35896,13030],{"mathvariant":1053},[1870,35898,35899,35901,35903],{},[371,35900,12890],{},[371,35902,612],{},[371,35904,1057],{},[738,35906,1092],{"stretchy":1077},[355,35908,35909,35919,35921,35929,35931,35939],{},[368,35910,35911,35913],{},[371,35912,5685],{},[742,35914,35915,35917],{},[371,35916,14101],{},[738,35918,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,35920,1078],{"stretchy":1077},[1870,35922,35923,35925,35927],{},[371,35924,38],{},[371,35926,612],{},[371,35928,1057],{},[371,35930,13030],{"mathvariant":1053},[1870,35932,35933,35935,35937],{},[371,35934,12890],{},[371,35936,612],{},[371,35938,1057],{},[738,35940,1092],{"stretchy":1077},[358,35942,35943,35975],{},[355,35944,35945,35947,35953,35955,35963,35965,35973],{},[371,35946,14525],{"mathvariant":1053},[368,35948,35949,35951],{},[371,35950,5685],{},[371,35952,14101],{},[738,35954,1078],{"stretchy":1077},[1870,35956,35957,35959,35961],{},[371,35958,38],{},[371,35960,612],{},[371,35962,1057],{},[371,35964,13030],{"mathvariant":1053},[1870,35966,35967,35969,35971],{},[371,35968,12890],{},[371,35970,612],{},[371,35972,1057],{},[738,35974,1092],{"stretchy":1077},[355,35976,35977,35983,35985,35993,35995,36003],{},[368,35978,35979,35981],{},[371,35980,5685],{},[371,35982,14101],{},[738,35984,1078],{"stretchy":1077},[1870,35986,35987,35989,35991],{},[371,35988,38],{},[371,35990,612],{},[371,35992,1057],{},[371,35994,13030],{"mathvariant":1053},[1870,35996,35997,35999,36001],{},[371,35998,12890],{},[371,36000,612],{},[371,36002,1057],{},[738,36004,1092],{"stretchy":1077},[14513,36006,36007,36013],{},[14516,36008,36009],{},[14519,36010,36011],{"scriptlevel":1802,"displaystyle":386},[355,36012],{},[14516,36014,36015],{},[14519,36016,36017],{"scriptlevel":1802,"displaystyle":386},[355,36018,36019,36021,36023,36029,36043,36061,36079,36081,36089,36091,36099,36101],{},[355,36020],{},[738,36022,204],{},[358,36024,36025,36027],{},[361,36026,363],{},[371,36028,7342],{},[14828,36030,36031,36033,36041],{},[738,36032,1874],{},[355,36034,36035,36037,36039],{},[371,36036,612],{},[738,36038,204],{},[361,36040,363],{},[371,36042,7342],{},[14828,36044,36045,36047,36055],{},[738,36046,1874],{},[355,36048,36049,36051,36053],{},[371,36050,1057],{},[738,36052,204],{},[361,36054,363],{},[368,36056,36057,36059],{},[371,36058,13358],{},[371,36060,612],{},[1870,36062,36063,36065,36071],{},[371,36064,1054],{},[742,36066,36067,36069],{},[371,36068,14101],{},[738,36070,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,36072,36073,36075,36077],{},[371,36074,27970],{},[371,36076,1054],{},[371,36078,5688],{},[738,36080,1078],{"stretchy":1077},[1870,36082,36083,36085,36087],{},[371,36084,12890],{},[371,36086,612],{},[371,36088,1057],{},[738,36090,1083],{"separator":386},[1870,36092,36093,36095,36097],{},[371,36094,38],{},[371,36096,612],{},[371,36098,1057],{},[738,36100,1092],{"stretchy":1077},[358,36102,36103,36135],{},[355,36104,36105,36107,36113,36115,36123,36125,36133],{},[371,36106,14525],{"mathvariant":1053},[368,36108,36109,36111],{},[371,36110,5685],{},[371,36112,14101],{},[738,36114,1078],{"stretchy":1077},[1870,36116,36117,36119,36121],{},[371,36118,38],{},[371,36120,612],{},[371,36122,1057],{},[371,36124,13030],{"mathvariant":1053},[1870,36126,36127,36129,36131],{},[371,36128,12890],{},[371,36130,612],{},[371,36132,1057],{},[738,36134,1092],{"stretchy":1077},[355,36136,36137,36147,36149,36157,36159,36167],{},[368,36138,36139,36141],{},[371,36140,5685],{},[742,36142,36143,36145],{},[371,36144,14101],{},[738,36146,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,36148,1078],{"stretchy":1077},[1870,36150,36151,36153,36155],{},[371,36152,38],{},[371,36154,612],{},[371,36156,1057],{},[371,36158,13030],{"mathvariant":1053},[1870,36160,36161,36163,36165],{},[371,36162,12890],{},[371,36164,612],{},[371,36166,1057],{},[738,36168,1092],{"stretchy":1077},[378,36170,36171],{"encoding":380},"\\begin{aligned}\n&\\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_\\theta^{GAE}(s_n^t, a_n^t) \\nabla \\log P_\\theta(a_n^t | s_n^t) \\\\\n&= \\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{P_\\theta(a_n^t | s_n^t)}{P_{\\theta'}(a_n^t | s_n^t)} \\nabla \\log P_\\theta(a_n^t | s_n^t) \\\\\n&= \\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{P_\\theta(a_n^t | s_n^t)}{P_{\\theta'}(a_n^t | s_n^t)} \\frac{\\nabla P_\\theta(a_n^t | s_n^t)}{P_\\theta(a_n^t | s_n^t)} \\\\\n&= \\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{\\nabla P_\\theta(a_n^t | s_n^t)}{P_{\\theta'}(a_n^t | s_n^t)}\n\\end{aligned}",[63,36173,36175],{"className":36174,"ariaHidden":386},[385],[63,36176,36178,36182],{"className":36177},[390],[63,36179],{"className":36180,"style":36181},[394],"height:13.6262em;vertical-align:-6.5631em;",[63,36183,36185],{"className":36184},[399],[63,36186,36188,36249],{"className":36187},[14507],[63,36189,36191],{"className":36190},[15298],[63,36192,36194,36240],{"className":36193},[411,412],[63,36195,36197,36237],{"className":36196},[416],[63,36198,36201,36210,36219,36228],{"className":36199,"style":36200},[420],"height:7.0631em;",[63,36202,36204,36207],{"style":36203},"top:-9.0631em;",[63,36205],{"className":36206,"style":15315},[428],[63,36208],{"className":36209},[399],[63,36211,36213,36216],{"style":36212},"top:-5.6565em;",[63,36214],{"className":36215,"style":15315},[428],[63,36217],{"className":36218},[399],[63,36220,36222,36225],{"style":36221},"top:-2.25em;",[63,36223],{"className":36224,"style":15315},[428],[63,36226],{"className":36227},[399],[63,36229,36231,36234],{"style":36230},"top:1.1565em;",[63,36232],{"className":36233,"style":15315},[428],[63,36235],{"className":36236},[399],[63,36238,507],{"className":36239},[506],[63,36241,36243],{"className":36242},[416],[63,36244,36247],{"className":36245,"style":36246},[420],"height:6.5631em;",[63,36248],{},[63,36250,36252],{"className":36251},[15522],[63,36253,36255,39961],{"className":36254},[411,412],[63,36256,36258,39958],{"className":36257},[416],[63,36259,36261,36853,37878,39097],{"className":36260,"style":36200},[420],[63,36262,36263,36266],{"style":36203},[63,36264],{"className":36265,"style":15315},[428],[63,36267,36269,36272,36334,36337,36404,36407,36511,36514,36574,36577,36628,36631,36634,36685,36688,36691,36694,36699,36702,36742,36745,36796,36799,36850],{"className":36268},[399],[63,36270],{"className":36271},[399],[63,36273,36275,36278,36331],{"className":36274},[399],[63,36276],{"className":36277},[403,404],[63,36279,36281],{"className":36280},[358],[63,36282,36284,36323],{"className":36283},[411,412],[63,36285,36287,36320],{"className":36286},[416],[63,36288,36290,36301,36309],{"className":36289,"style":16424},[420],[63,36291,36292,36295],{"style":15964},[63,36293],{"className":36294,"style":429},[428],[63,36296,36298],{"className":36297},[399],[63,36299,7342],{"className":36300,"style":16436},[399,472],[63,36302,36303,36306],{"style":554},[63,36304],{"className":36305,"style":429},[428],[63,36307],{"className":36308,"style":562},[561],[63,36310,36311,36314],{"style":16030},[63,36312],{"className":36313,"style":429},[428],[63,36315,36317],{"className":36316},[399],[63,36318,363],{"className":36319},[399],[63,36321,507],{"className":36322},[506],[63,36324,36326],{"className":36325},[416],[63,36327,36329],{"className":36328,"style":16465},[420],[63,36330],{},[63,36332],{"className":36333},[593,404],[63,36335],{"className":36336,"style":1177},[776],[63,36338,36340],{"className":36339},[1212,14383],[63,36341,36343,36396],{"className":36342},[411,412],[63,36344,36346,36393],{"className":36345},[416],[63,36347,36349,36369,36379],{"className":36348,"style":16486},[420],[63,36350,36351,36354],{"style":16489},[63,36352],{"className":36353,"style":15577},[428],[63,36355,36357],{"className":36356},[433,434,435,436],[63,36358,36360,36363,36366],{"className":36359},[399,436],[63,36361,612],{"className":36362},[399,472,436],[63,36364,204],{"className":36365},[781,436],[63,36367,363],{"className":36368},[399,436],[63,36370,36371,36374],{"style":15586},[63,36372],{"className":36373,"style":15577},[428],[63,36375,36376],{},[63,36377,1874],{"className":36378},[1212,1958,15595],[63,36380,36381,36384],{"style":16520},[63,36382],{"className":36383,"style":15577},[428],[63,36385,36387],{"className":36386},[433,434,435,436],[63,36388,36390],{"className":36389},[399,436],[63,36391,7342],{"className":36392,"style":16436},[399,472,436],[63,36394,507],{"className":36395},[506],[63,36397,36399],{"className":36398},[416],[63,36400,36402],{"className":36401,"style":16539},[420],[63,36403],{},[63,36405],{"className":36406,"style":1177},[776],[63,36408,36410],{"className":36409},[1212,14383],[63,36411,36413,36503],{"className":36412},[411,412],[63,36414,36416,36500],{"className":36415},[416],[63,36417,36419,36439,36449],{"className":36418,"style":17295},[420],[63,36420,36421,36424],{"style":16489},[63,36422],{"className":36423,"style":15577},[428],[63,36425,36427],{"className":36426},[433,434,435,436],[63,36428,36430,36433,36436],{"className":36429},[399,436],[63,36431,1057],{"className":36432},[399,472,436],[63,36434,204],{"className":36435},[781,436],[63,36437,363],{"className":36438},[399,436],[63,36440,36441,36444],{"style":15586},[63,36442],{"className":36443,"style":15577},[428],[63,36445,36446],{},[63,36447,1874],{"className":36448},[1212,1958,15595],[63,36450,36451,36454],{"style":17328},[63,36452],{"className":36453,"style":15577},[428],[63,36455,36457],{"className":36456},[433,434,435,436],[63,36458,36460],{"className":36459},[399,436],[63,36461,36463,36466],{"className":36462},[399,436],[63,36464,13358],{"className":36465,"style":3251},[399,472,436],[63,36467,36469],{"className":36468},[476],[63,36470,36472,36492],{"className":36471},[411,412],[63,36473,36475,36489],{"className":36474},[416],[63,36476,36478],{"className":36477,"style":4367},[420],[63,36479,36480,36483],{"style":17358},[63,36481],{"className":36482,"style":493},[428],[63,36484,36486],{"className":36485},[433,497,498,436],[63,36487,612],{"className":36488},[399,472,436],[63,36490,507],{"className":36491},[506],[63,36493,36495],{"className":36494},[416],[63,36496,36498],{"className":36497,"style":4389},[420],[63,36499],{},[63,36501,507],{"className":36502},[506],[63,36504,36506],{"className":36505},[416],[63,36507,36509],{"className":36508,"style":16539},[420],[63,36510],{},[63,36512],{"className":36513,"style":1177},[776],[63,36515,36517,36520],{"className":36516},[399],[63,36518,1054],{"className":36519},[399,472],[63,36521,36523],{"className":36522},[476],[63,36524,36526,36566],{"className":36525},[411,412],[63,36527,36529,36563],{"className":36528},[416],[63,36530,36532,36543],{"className":36531,"style":25181},[420],[63,36533,36534,36537],{"style":17457},[63,36535],{"className":36536,"style":817},[428],[63,36538,36540],{"className":36539},[433,434,435,436],[63,36541,14101],{"className":36542,"style":3276},[399,472,436],[63,36544,36545,36548],{"style":16572},[63,36546],{"className":36547,"style":817},[428],[63,36549,36551],{"className":36550},[433,434,435,436],[63,36552,36554,36557,36560],{"className":36553},[399,436],[63,36555,27970],{"className":36556},[399,472,436],[63,36558,1054],{"className":36559},[399,472,436],[63,36561,5688],{"className":36562,"style":6101},[399,472,436],[63,36564,507],{"className":36565},[506],[63,36567,36569],{"className":36568},[416],[63,36570,36572],{"className":36571,"style":17487},[420],[63,36573],{},[63,36575,1078],{"className":36576},[403],[63,36578,36580,36583],{"className":36579},[399],[63,36581,12890],{"className":36582},[399,472],[63,36584,36586],{"className":36585},[476],[63,36587,36589,36620],{"className":36588},[411,412],[63,36590,36592,36617],{"className":36591},[416],[63,36593,36595,36606],{"className":36594,"style":17454},[420],[63,36596,36597,36600],{"style":17457},[63,36598],{"className":36599,"style":817},[428],[63,36601,36603],{"className":36602},[433,434,435,436],[63,36604,612],{"className":36605},[399,472,436],[63,36607,36608,36611],{"style":16572},[63,36609],{"className":36610,"style":817},[428],[63,36612,36614],{"className":36613},[433,434,435,436],[63,36615,1057],{"className":36616},[399,472,436],[63,36618,507],{"className":36619},[506],[63,36621,36623],{"className":36622},[416],[63,36624,36626],{"className":36625,"style":17487},[420],[63,36627],{},[63,36629,1083],{"className":36630},[1173],[63,36632],{"className":36633,"style":1177},[776],[63,36635,36637,36640],{"className":36636},[399],[63,36638,38],{"className":36639},[399,472],[63,36641,36643],{"className":36642},[476],[63,36644,36646,36677],{"className":36645},[411,412],[63,36647,36649,36674],{"className":36648},[416],[63,36650,36652,36663],{"className":36651,"style":17454},[420],[63,36653,36654,36657],{"style":17457},[63,36655],{"className":36656,"style":817},[428],[63,36658,36660],{"className":36659},[433,434,435,436],[63,36661,612],{"className":36662},[399,472,436],[63,36664,36665,36668],{"style":16572},[63,36666],{"className":36667,"style":817},[428],[63,36669,36671],{"className":36670},[433,434,435,436],[63,36672,1057],{"className":36673},[399,472,436],[63,36675,507],{"className":36676},[506],[63,36678,36680],{"className":36679},[416],[63,36681,36683],{"className":36682,"style":17487},[420],[63,36684],{},[63,36686,1092],{"className":36687},[593],[63,36689,14525],{"className":36690},[399],[63,36692],{"className":36693,"style":1177},[776],[63,36695,16996,36697],{"className":36696},[1212],[63,36698,17000],{"style":16999},[63,36700],{"className":36701,"style":1177},[776],[63,36703,36705,36708],{"className":36704},[399],[63,36706,5685],{"className":36707,"style":3251},[399,472],[63,36709,36711],{"className":36710},[476],[63,36712,36714,36734],{"className":36713},[411,412],[63,36715,36717,36731],{"className":36716},[416],[63,36718,36720],{"className":36719,"style":1532},[420],[63,36721,36722,36725],{"style":7483},[63,36723],{"className":36724,"style":817},[428],[63,36726,36728],{"className":36727},[433,434,435,436],[63,36729,14101],{"className":36730,"style":3276},[399,472,436],[63,36732,507],{"className":36733},[506],[63,36735,36737],{"className":36736},[416],[63,36738,36740],{"className":36739,"style":1554},[420],[63,36741],{},[63,36743,1078],{"className":36744},[403],[63,36746,36748,36751],{"className":36747},[399],[63,36749,38],{"className":36750},[399,472],[63,36752,36754],{"className":36753},[476],[63,36755,36757,36788],{"className":36756},[411,412],[63,36758,36760,36785],{"className":36759},[416],[63,36761,36763,36774],{"className":36762,"style":17454},[420],[63,36764,36765,36768],{"style":17457},[63,36766],{"className":36767,"style":817},[428],[63,36769,36771],{"className":36770},[433,434,435,436],[63,36772,612],{"className":36773},[399,472,436],[63,36775,36776,36779],{"style":16572},[63,36777],{"className":36778,"style":817},[428],[63,36780,36782],{"className":36781},[433,434,435,436],[63,36783,1057],{"className":36784},[399,472,436],[63,36786,507],{"className":36787},[506],[63,36789,36791],{"className":36790},[416],[63,36792,36794],{"className":36793,"style":17487},[420],[63,36795],{},[63,36797,13030],{"className":36798},[399],[63,36800,36802,36805],{"className":36801},[399],[63,36803,12890],{"className":36804},[399,472],[63,36806,36808],{"className":36807},[476],[63,36809,36811,36842],{"className":36810},[411,412],[63,36812,36814,36839],{"className":36813},[416],[63,36815,36817,36828],{"className":36816,"style":17454},[420],[63,36818,36819,36822],{"style":17457},[63,36820],{"className":36821,"style":817},[428],[63,36823,36825],{"className":36824},[433,434,435,436],[63,36826,612],{"className":36827},[399,472,436],[63,36829,36830,36833],{"style":16572},[63,36831],{"className":36832,"style":817},[428],[63,36834,36836],{"className":36835},[433,434,435,436],[63,36837,1057],{"className":36838},[399,472,436],[63,36840,507],{"className":36841},[506],[63,36843,36845],{"className":36844},[416],[63,36846,36848],{"className":36847,"style":17487},[420],[63,36849],{},[63,36851,1092],{"className":36852},[593],[63,36854,36855,36858],{"style":36212},[63,36856],{"className":36857,"style":15315},[428],[63,36859,36861,36864,36867,36870,36873,36935,36938,37005,37008,37112,37115,37207,37210,37261,37264,37267,37318,37321,37713,37716,37719,37724,37727,37767,37770,37821,37824,37875],{"className":36860},[399],[63,36862],{"className":36863},[399],[63,36865],{"className":36866,"style":777},[776],[63,36868,204],{"className":36869},[781],[63,36871],{"className":36872,"style":777},[776],[63,36874,36876,36879,36932],{"className":36875},[399],[63,36877],{"className":36878},[403,404],[63,36880,36882],{"className":36881},[358],[63,36883,36885,36924],{"className":36884},[411,412],[63,36886,36888,36921],{"className":36887},[416],[63,36889,36891,36902,36910],{"className":36890,"style":16424},[420],[63,36892,36893,36896],{"style":15964},[63,36894],{"className":36895,"style":429},[428],[63,36897,36899],{"className":36898},[399],[63,36900,7342],{"className":36901,"style":16436},[399,472],[63,36903,36904,36907],{"style":554},[63,36905],{"className":36906,"style":429},[428],[63,36908],{"className":36909,"style":562},[561],[63,36911,36912,36915],{"style":16030},[63,36913],{"className":36914,"style":429},[428],[63,36916,36918],{"className":36917},[399],[63,36919,363],{"className":36920},[399],[63,36922,507],{"className":36923},[506],[63,36925,36927],{"className":36926},[416],[63,36928,36930],{"className":36929,"style":16465},[420],[63,36931],{},[63,36933],{"className":36934},[593,404],[63,36936],{"className":36937,"style":1177},[776],[63,36939,36941],{"className":36940},[1212,14383],[63,36942,36944,36997],{"className":36943},[411,412],[63,36945,36947,36994],{"className":36946},[416],[63,36948,36950,36970,36980],{"className":36949,"style":16486},[420],[63,36951,36952,36955],{"style":16489},[63,36953],{"className":36954,"style":15577},[428],[63,36956,36958],{"className":36957},[433,434,435,436],[63,36959,36961,36964,36967],{"className":36960},[399,436],[63,36962,612],{"className":36963},[399,472,436],[63,36965,204],{"className":36966},[781,436],[63,36968,363],{"className":36969},[399,436],[63,36971,36972,36975],{"style":15586},[63,36973],{"className":36974,"style":15577},[428],[63,36976,36977],{},[63,36978,1874],{"className":36979},[1212,1958,15595],[63,36981,36982,36985],{"style":16520},[63,36983],{"className":36984,"style":15577},[428],[63,36986,36988],{"className":36987},[433,434,435,436],[63,36989,36991],{"className":36990},[399,436],[63,36992,7342],{"className":36993,"style":16436},[399,472,436],[63,36995,507],{"className":36996},[506],[63,36998,37000],{"className":36999},[416],[63,37001,37003],{"className":37002,"style":16539},[420],[63,37004],{},[63,37006],{"className":37007,"style":1177},[776],[63,37009,37011],{"className":37010},[1212,14383],[63,37012,37014,37104],{"className":37013},[411,412],[63,37015,37017,37101],{"className":37016},[416],[63,37018,37020,37040,37050],{"className":37019,"style":17295},[420],[63,37021,37022,37025],{"style":16489},[63,37023],{"className":37024,"style":15577},[428],[63,37026,37028],{"className":37027},[433,434,435,436],[63,37029,37031,37034,37037],{"className":37030},[399,436],[63,37032,1057],{"className":37033},[399,472,436],[63,37035,204],{"className":37036},[781,436],[63,37038,363],{"className":37039},[399,436],[63,37041,37042,37045],{"style":15586},[63,37043],{"className":37044,"style":15577},[428],[63,37046,37047],{},[63,37048,1874],{"className":37049},[1212,1958,15595],[63,37051,37052,37055],{"style":17328},[63,37053],{"className":37054,"style":15577},[428],[63,37056,37058],{"className":37057},[433,434,435,436],[63,37059,37061],{"className":37060},[399,436],[63,37062,37064,37067],{"className":37063},[399,436],[63,37065,13358],{"className":37066,"style":3251},[399,472,436],[63,37068,37070],{"className":37069},[476],[63,37071,37073,37093],{"className":37072},[411,412],[63,37074,37076,37090],{"className":37075},[416],[63,37077,37079],{"className":37078,"style":4367},[420],[63,37080,37081,37084],{"style":17358},[63,37082],{"className":37083,"style":493},[428],[63,37085,37087],{"className":37086},[433,497,498,436],[63,37088,612],{"className":37089},[399,472,436],[63,37091,507],{"className":37092},[506],[63,37094,37096],{"className":37095},[416],[63,37097,37099],{"className":37098,"style":4389},[420],[63,37100],{},[63,37102,507],{"className":37103},[506],[63,37105,37107],{"className":37106},[416],[63,37108,37110],{"className":37109,"style":16539},[420],[63,37111],{},[63,37113],{"className":37114,"style":1177},[776],[63,37116,37118,37121],{"className":37117},[399],[63,37119,1054],{"className":37120},[399,472],[63,37122,37124],{"className":37123},[476],[63,37125,37127,37199],{"className":37126},[411,412],[63,37128,37130,37196],{"className":37129},[416],[63,37131,37133,37176],{"className":37132,"style":25181},[420],[63,37134,37135,37138],{"style":17457},[63,37136],{"className":37137,"style":817},[428],[63,37139,37141],{"className":37140},[433,434,435,436],[63,37142,37144],{"className":37143},[399,436],[63,37145,37147,37150],{"className":37146},[399,436],[63,37148,14101],{"className":37149,"style":3276},[399,472,436],[63,37151,37153],{"className":37152},[476],[63,37154,37156],{"className":37155},[411],[63,37157,37159],{"className":37158},[416],[63,37160,37162],{"className":37161,"style":19291},[420],[63,37163,37164,37167],{"style":19294},[63,37165],{"className":37166,"style":493},[428],[63,37168,37170],{"className":37169},[433,497,498,436],[63,37171,37173],{"className":37172},[399,436],[63,37174,19131],{"className":37175},[399,436],[63,37177,37178,37181],{"style":16572},[63,37179],{"className":37180,"style":817},[428],[63,37182,37184],{"className":37183},[433,434,435,436],[63,37185,37187,37190,37193],{"className":37186},[399,436],[63,37188,27970],{"className":37189},[399,472,436],[63,37191,1054],{"className":37192},[399,472,436],[63,37194,5688],{"className":37195,"style":6101},[399,472,436],[63,37197,507],{"className":37198},[506],[63,37200,37202],{"className":37201},[416],[63,37203,37205],{"className":37204,"style":17487},[420],[63,37206],{},[63,37208,1078],{"className":37209},[403],[63,37211,37213,37216],{"className":37212},[399],[63,37214,12890],{"className":37215},[399,472],[63,37217,37219],{"className":37218},[476],[63,37220,37222,37253],{"className":37221},[411,412],[63,37223,37225,37250],{"className":37224},[416],[63,37226,37228,37239],{"className":37227,"style":17454},[420],[63,37229,37230,37233],{"style":17457},[63,37231],{"className":37232,"style":817},[428],[63,37234,37236],{"className":37235},[433,434,435,436],[63,37237,612],{"className":37238},[399,472,436],[63,37240,37241,37244],{"style":16572},[63,37242],{"className":37243,"style":817},[428],[63,37245,37247],{"className":37246},[433,434,435,436],[63,37248,1057],{"className":37249},[399,472,436],[63,37251,507],{"className":37252},[506],[63,37254,37256],{"className":37255},[416],[63,37257,37259],{"className":37258,"style":17487},[420],[63,37260],{},[63,37262,1083],{"className":37263},[1173],[63,37265],{"className":37266,"style":1177},[776],[63,37268,37270,37273],{"className":37269},[399],[63,37271,38],{"className":37272},[399,472],[63,37274,37276],{"className":37275},[476],[63,37277,37279,37310],{"className":37278},[411,412],[63,37280,37282,37307],{"className":37281},[416],[63,37283,37285,37296],{"className":37284,"style":17454},[420],[63,37286,37287,37290],{"style":17457},[63,37288],{"className":37289,"style":817},[428],[63,37291,37293],{"className":37292},[433,434,435,436],[63,37294,612],{"className":37295},[399,472,436],[63,37297,37298,37301],{"style":16572},[63,37299],{"className":37300,"style":817},[428],[63,37302,37304],{"className":37303},[433,434,435,436],[63,37305,1057],{"className":37306},[399,472,436],[63,37308,507],{"className":37309},[506],[63,37311,37313],{"className":37312},[416],[63,37314,37316],{"className":37315,"style":17487},[420],[63,37317],{},[63,37319,1092],{"className":37320},[593],[63,37322,37324,37327,37710],{"className":37323},[399],[63,37325],{"className":37326},[403,404],[63,37328,37330],{"className":37329},[358],[63,37331,37333,37702],{"className":37332},[411,412],[63,37334,37336,37699],{"className":37335},[416],[63,37337,37340,37532,37540],{"className":37338,"style":37339},[420],"height:1.4706em;",[63,37341,37342,37345],{"style":15964},[63,37343],{"className":37344,"style":429},[428],[63,37346,37348,37420,37423,37475,37478,37529],{"className":37347},[399],[63,37349,37351,37354],{"className":37350},[399],[63,37352,5685],{"className":37353,"style":3251},[399,472],[63,37355,37357],{"className":37356},[476],[63,37358,37360,37412],{"className":37359},[411,412],[63,37361,37363,37409],{"className":37362},[416],[63,37364,37366],{"className":37365,"style":1532},[420],[63,37367,37368,37371],{"style":7483},[63,37369],{"className":37370,"style":817},[428],[63,37372,37374],{"className":37373},[433,434,435,436],[63,37375,37377],{"className":37376},[399,436],[63,37378,37380,37383],{"className":37379},[399,436],[63,37381,14101],{"className":37382,"style":3276},[399,472,436],[63,37384,37386],{"className":37385},[476],[63,37387,37389],{"className":37388},[411],[63,37390,37392],{"className":37391},[416],[63,37393,37395],{"className":37394,"style":19291},[420],[63,37396,37397,37400],{"style":19294},[63,37398],{"className":37399,"style":493},[428],[63,37401,37403],{"className":37402},[433,497,498,436],[63,37404,37406],{"className":37405},[399,436],[63,37407,19131],{"className":37408},[399,436],[63,37410,507],{"className":37411},[506],[63,37413,37415],{"className":37414},[416],[63,37416,37418],{"className":37417,"style":1554},[420],[63,37419],{},[63,37421,1078],{"className":37422},[403],[63,37424,37426,37429],{"className":37425},[399],[63,37427,38],{"className":37428},[399,472],[63,37430,37432],{"className":37431},[476],[63,37433,37435,37467],{"className":37434},[411,412],[63,37436,37438,37464],{"className":37437},[416],[63,37439,37442,37453],{"className":37440,"style":37441},[420],"height:0.7196em;",[63,37443,37444,37447],{"style":17457},[63,37445],{"className":37446,"style":817},[428],[63,37448,37450],{"className":37449},[433,434,435,436],[63,37451,612],{"className":37452},[399,472,436],[63,37454,37455,37458],{"style":16675},[63,37456],{"className":37457,"style":817},[428],[63,37459,37461],{"className":37460},[433,434,435,436],[63,37462,1057],{"className":37463},[399,472,436],[63,37465,507],{"className":37466},[506],[63,37468,37470],{"className":37469},[416],[63,37471,37473],{"className":37472,"style":17487},[420],[63,37474],{},[63,37476,13030],{"className":37477},[399],[63,37479,37481,37484],{"className":37480},[399],[63,37482,12890],{"className":37483},[399,472],[63,37485,37487],{"className":37486},[476],[63,37488,37490,37521],{"className":37489},[411,412],[63,37491,37493,37518],{"className":37492},[416],[63,37494,37496,37507],{"className":37495,"style":37441},[420],[63,37497,37498,37501],{"style":17457},[63,37499],{"className":37500,"style":817},[428],[63,37502,37504],{"className":37503},[433,434,435,436],[63,37505,612],{"className":37506},[399,472,436],[63,37508,37509,37512],{"style":16675},[63,37510],{"className":37511,"style":817},[428],[63,37513,37515],{"className":37514},[433,434,435,436],[63,37516,1057],{"className":37517},[399,472,436],[63,37519,507],{"className":37520},[506],[63,37522,37524],{"className":37523},[416],[63,37525,37527],{"className":37526,"style":17487},[420],[63,37528],{},[63,37530,1092],{"className":37531},[593],[63,37533,37534,37537],{"style":554},[63,37535],{"className":37536,"style":429},[428],[63,37538],{"className":37539,"style":562},[561],[63,37541,37542,37545],{"style":16030},[63,37543],{"className":37544,"style":429},[428],[63,37546,37548,37588,37591,37642,37645,37696],{"className":37547},[399],[63,37549,37551,37554],{"className":37550},[399],[63,37552,5685],{"className":37553,"style":3251},[399,472],[63,37555,37557],{"className":37556},[476],[63,37558,37560,37580],{"className":37559},[411,412],[63,37561,37563,37577],{"className":37562},[416],[63,37564,37566],{"className":37565,"style":1532},[420],[63,37567,37568,37571],{"style":7483},[63,37569],{"className":37570,"style":817},[428],[63,37572,37574],{"className":37573},[433,434,435,436],[63,37575,14101],{"className":37576,"style":3276},[399,472,436],[63,37578,507],{"className":37579},[506],[63,37581,37583],{"className":37582},[416],[63,37584,37586],{"className":37585,"style":1554},[420],[63,37587],{},[63,37589,1078],{"className":37590},[403],[63,37592,37594,37597],{"className":37593},[399],[63,37595,38],{"className":37596},[399,472],[63,37598,37600],{"className":37599},[476],[63,37601,37603,37634],{"className":37602},[411,412],[63,37604,37606,37631],{"className":37605},[416],[63,37607,37609,37620],{"className":37608,"style":18996},[420],[63,37610,37611,37614],{"style":17457},[63,37612],{"className":37613,"style":817},[428],[63,37615,37617],{"className":37616},[433,434,435,436],[63,37618,612],{"className":37619},[399,472,436],[63,37621,37622,37625],{"style":813},[63,37623],{"className":37624,"style":817},[428],[63,37626,37628],{"className":37627},[433,434,435,436],[63,37629,1057],{"className":37630},[399,472,436],[63,37632,507],{"className":37633},[506],[63,37635,37637],{"className":37636},[416],[63,37638,37640],{"className":37639,"style":17487},[420],[63,37641],{},[63,37643,13030],{"className":37644},[399],[63,37646,37648,37651],{"className":37647},[399],[63,37649,12890],{"className":37650},[399,472],[63,37652,37654],{"className":37653},[476],[63,37655,37657,37688],{"className":37656},[411,412],[63,37658,37660,37685],{"className":37659},[416],[63,37661,37663,37674],{"className":37662,"style":18996},[420],[63,37664,37665,37668],{"style":17457},[63,37666],{"className":37667,"style":817},[428],[63,37669,37671],{"className":37670},[433,434,435,436],[63,37672,612],{"className":37673},[399,472,436],[63,37675,37676,37679],{"style":813},[63,37677],{"className":37678,"style":817},[428],[63,37680,37682],{"className":37681},[433,434,435,436],[63,37683,1057],{"className":37684},[399,472,436],[63,37686,507],{"className":37687},[506],[63,37689,37691],{"className":37690},[416],[63,37692,37694],{"className":37693,"style":17487},[420],[63,37695],{},[63,37697,1092],{"className":37698},[593],[63,37700,507],{"className":37701},[506],[63,37703,37705],{"className":37704},[416],[63,37706,37708],{"className":37707,"style":16095},[420],[63,37709],{},[63,37711],{"className":37712},[593,404],[63,37714,14525],{"className":37715},[399],[63,37717],{"className":37718,"style":1177},[776],[63,37720,16996,37722],{"className":37721},[1212],[63,37723,17000],{"style":16999},[63,37725],{"className":37726,"style":1177},[776],[63,37728,37730,37733],{"className":37729},[399],[63,37731,5685],{"className":37732,"style":3251},[399,472],[63,37734,37736],{"className":37735},[476],[63,37737,37739,37759],{"className":37738},[411,412],[63,37740,37742,37756],{"className":37741},[416],[63,37743,37745],{"className":37744,"style":1532},[420],[63,37746,37747,37750],{"style":7483},[63,37748],{"className":37749,"style":817},[428],[63,37751,37753],{"className":37752},[433,434,435,436],[63,37754,14101],{"className":37755,"style":3276},[399,472,436],[63,37757,507],{"className":37758},[506],[63,37760,37762],{"className":37761},[416],[63,37763,37765],{"className":37764,"style":1554},[420],[63,37766],{},[63,37768,1078],{"className":37769},[403],[63,37771,37773,37776],{"className":37772},[399],[63,37774,38],{"className":37775},[399,472],[63,37777,37779],{"className":37778},[476],[63,37780,37782,37813],{"className":37781},[411,412],[63,37783,37785,37810],{"className":37784},[416],[63,37786,37788,37799],{"className":37787,"style":17454},[420],[63,37789,37790,37793],{"style":17457},[63,37791],{"className":37792,"style":817},[428],[63,37794,37796],{"className":37795},[433,434,435,436],[63,37797,612],{"className":37798},[399,472,436],[63,37800,37801,37804],{"style":16572},[63,37802],{"className":37803,"style":817},[428],[63,37805,37807],{"className":37806},[433,434,435,436],[63,37808,1057],{"className":37809},[399,472,436],[63,37811,507],{"className":37812},[506],[63,37814,37816],{"className":37815},[416],[63,37817,37819],{"className":37818,"style":17487},[420],[63,37820],{},[63,37822,13030],{"className":37823},[399],[63,37825,37827,37830],{"className":37826},[399],[63,37828,12890],{"className":37829},[399,472],[63,37831,37833],{"className":37832},[476],[63,37834,37836,37867],{"className":37835},[411,412],[63,37837,37839,37864],{"className":37838},[416],[63,37840,37842,37853],{"className":37841,"style":17454},[420],[63,37843,37844,37847],{"style":17457},[63,37845],{"className":37846,"style":817},[428],[63,37848,37850],{"className":37849},[433,434,435,436],[63,37851,612],{"className":37852},[399,472,436],[63,37854,37855,37858],{"style":16572},[63,37856],{"className":37857,"style":817},[428],[63,37859,37861],{"className":37860},[433,434,435,436],[63,37862,1057],{"className":37863},[399,472,436],[63,37865,507],{"className":37866},[506],[63,37868,37870],{"className":37869},[416],[63,37871,37873],{"className":37872,"style":17487},[420],[63,37874],{},[63,37876,1092],{"className":37877},[593],[63,37879,37880,37883],{"style":36221},[63,37881],{"className":37882,"style":15315},[428],[63,37884,37886,37889,37892,37895,37898,37960,37963,38030,38033,38137,38140,38232,38235,38286,38289,38292,38343,38346,38736],{"className":37885},[399],[63,37887],{"className":37888},[399],[63,37890],{"className":37891,"style":777},[776],[63,37893,204],{"className":37894},[781],[63,37896],{"className":37897,"style":777},[776],[63,37899,37901,37904,37957],{"className":37900},[399],[63,37902],{"className":37903},[403,404],[63,37905,37907],{"className":37906},[358],[63,37908,37910,37949],{"className":37909},[411,412],[63,37911,37913,37946],{"className":37912},[416],[63,37914,37916,37927,37935],{"className":37915,"style":16424},[420],[63,37917,37918,37921],{"style":15964},[63,37919],{"className":37920,"style":429},[428],[63,37922,37924],{"className":37923},[399],[63,37925,7342],{"className":37926,"style":16436},[399,472],[63,37928,37929,37932],{"style":554},[63,37930],{"className":37931,"style":429},[428],[63,37933],{"className":37934,"style":562},[561],[63,37936,37937,37940],{"style":16030},[63,37938],{"className":37939,"style":429},[428],[63,37941,37943],{"className":37942},[399],[63,37944,363],{"className":37945},[399],[63,37947,507],{"className":37948},[506],[63,37950,37952],{"className":37951},[416],[63,37953,37955],{"className":37954,"style":16465},[420],[63,37956],{},[63,37958],{"className":37959},[593,404],[63,37961],{"className":37962,"style":1177},[776],[63,37964,37966],{"className":37965},[1212,14383],[63,37967,37969,38022],{"className":37968},[411,412],[63,37970,37972,38019],{"className":37971},[416],[63,37973,37975,37995,38005],{"className":37974,"style":16486},[420],[63,37976,37977,37980],{"style":16489},[63,37978],{"className":37979,"style":15577},[428],[63,37981,37983],{"className":37982},[433,434,435,436],[63,37984,37986,37989,37992],{"className":37985},[399,436],[63,37987,612],{"className":37988},[399,472,436],[63,37990,204],{"className":37991},[781,436],[63,37993,363],{"className":37994},[399,436],[63,37996,37997,38000],{"style":15586},[63,37998],{"className":37999,"style":15577},[428],[63,38001,38002],{},[63,38003,1874],{"className":38004},[1212,1958,15595],[63,38006,38007,38010],{"style":16520},[63,38008],{"className":38009,"style":15577},[428],[63,38011,38013],{"className":38012},[433,434,435,436],[63,38014,38016],{"className":38015},[399,436],[63,38017,7342],{"className":38018,"style":16436},[399,472,436],[63,38020,507],{"className":38021},[506],[63,38023,38025],{"className":38024},[416],[63,38026,38028],{"className":38027,"style":16539},[420],[63,38029],{},[63,38031],{"className":38032,"style":1177},[776],[63,38034,38036],{"className":38035},[1212,14383],[63,38037,38039,38129],{"className":38038},[411,412],[63,38040,38042,38126],{"className":38041},[416],[63,38043,38045,38065,38075],{"className":38044,"style":17295},[420],[63,38046,38047,38050],{"style":16489},[63,38048],{"className":38049,"style":15577},[428],[63,38051,38053],{"className":38052},[433,434,435,436],[63,38054,38056,38059,38062],{"className":38055},[399,436],[63,38057,1057],{"className":38058},[399,472,436],[63,38060,204],{"className":38061},[781,436],[63,38063,363],{"className":38064},[399,436],[63,38066,38067,38070],{"style":15586},[63,38068],{"className":38069,"style":15577},[428],[63,38071,38072],{},[63,38073,1874],{"className":38074},[1212,1958,15595],[63,38076,38077,38080],{"style":17328},[63,38078],{"className":38079,"style":15577},[428],[63,38081,38083],{"className":38082},[433,434,435,436],[63,38084,38086],{"className":38085},[399,436],[63,38087,38089,38092],{"className":38088},[399,436],[63,38090,13358],{"className":38091,"style":3251},[399,472,436],[63,38093,38095],{"className":38094},[476],[63,38096,38098,38118],{"className":38097},[411,412],[63,38099,38101,38115],{"className":38100},[416],[63,38102,38104],{"className":38103,"style":4367},[420],[63,38105,38106,38109],{"style":17358},[63,38107],{"className":38108,"style":493},[428],[63,38110,38112],{"className":38111},[433,497,498,436],[63,38113,612],{"className":38114},[399,472,436],[63,38116,507],{"className":38117},[506],[63,38119,38121],{"className":38120},[416],[63,38122,38124],{"className":38123,"style":4389},[420],[63,38125],{},[63,38127,507],{"className":38128},[506],[63,38130,38132],{"className":38131},[416],[63,38133,38135],{"className":38134,"style":16539},[420],[63,38136],{},[63,38138],{"className":38139,"style":1177},[776],[63,38141,38143,38146],{"className":38142},[399],[63,38144,1054],{"className":38145},[399,472],[63,38147,38149],{"className":38148},[476],[63,38150,38152,38224],{"className":38151},[411,412],[63,38153,38155,38221],{"className":38154},[416],[63,38156,38158,38201],{"className":38157,"style":25181},[420],[63,38159,38160,38163],{"style":17457},[63,38161],{"className":38162,"style":817},[428],[63,38164,38166],{"className":38165},[433,434,435,436],[63,38167,38169],{"className":38168},[399,436],[63,38170,38172,38175],{"className":38171},[399,436],[63,38173,14101],{"className":38174,"style":3276},[399,472,436],[63,38176,38178],{"className":38177},[476],[63,38179,38181],{"className":38180},[411],[63,38182,38184],{"className":38183},[416],[63,38185,38187],{"className":38186,"style":19291},[420],[63,38188,38189,38192],{"style":19294},[63,38190],{"className":38191,"style":493},[428],[63,38193,38195],{"className":38194},[433,497,498,436],[63,38196,38198],{"className":38197},[399,436],[63,38199,19131],{"className":38200},[399,436],[63,38202,38203,38206],{"style":16572},[63,38204],{"className":38205,"style":817},[428],[63,38207,38209],{"className":38208},[433,434,435,436],[63,38210,38212,38215,38218],{"className":38211},[399,436],[63,38213,27970],{"className":38214},[399,472,436],[63,38216,1054],{"className":38217},[399,472,436],[63,38219,5688],{"className":38220,"style":6101},[399,472,436],[63,38222,507],{"className":38223},[506],[63,38225,38227],{"className":38226},[416],[63,38228,38230],{"className":38229,"style":17487},[420],[63,38231],{},[63,38233,1078],{"className":38234},[403],[63,38236,38238,38241],{"className":38237},[399],[63,38239,12890],{"className":38240},[399,472],[63,38242,38244],{"className":38243},[476],[63,38245,38247,38278],{"className":38246},[411,412],[63,38248,38250,38275],{"className":38249},[416],[63,38251,38253,38264],{"className":38252,"style":17454},[420],[63,38254,38255,38258],{"style":17457},[63,38256],{"className":38257,"style":817},[428],[63,38259,38261],{"className":38260},[433,434,435,436],[63,38262,612],{"className":38263},[399,472,436],[63,38265,38266,38269],{"style":16572},[63,38267],{"className":38268,"style":817},[428],[63,38270,38272],{"className":38271},[433,434,435,436],[63,38273,1057],{"className":38274},[399,472,436],[63,38276,507],{"className":38277},[506],[63,38279,38281],{"className":38280},[416],[63,38282,38284],{"className":38283,"style":17487},[420],[63,38285],{},[63,38287,1083],{"className":38288},[1173],[63,38290],{"className":38291,"style":1177},[776],[63,38293,38295,38298],{"className":38294},[399],[63,38296,38],{"className":38297},[399,472],[63,38299,38301],{"className":38300},[476],[63,38302,38304,38335],{"className":38303},[411,412],[63,38305,38307,38332],{"className":38306},[416],[63,38308,38310,38321],{"className":38309,"style":17454},[420],[63,38311,38312,38315],{"style":17457},[63,38313],{"className":38314,"style":817},[428],[63,38316,38318],{"className":38317},[433,434,435,436],[63,38319,612],{"className":38320},[399,472,436],[63,38322,38323,38326],{"style":16572},[63,38324],{"className":38325,"style":817},[428],[63,38327,38329],{"className":38328},[433,434,435,436],[63,38330,1057],{"className":38331},[399,472,436],[63,38333,507],{"className":38334},[506],[63,38336,38338],{"className":38337},[416],[63,38339,38341],{"className":38340,"style":17487},[420],[63,38342],{},[63,38344,1092],{"className":38345},[593],[63,38347,38349,38352,38733],{"className":38348},[399],[63,38350],{"className":38351},[403,404],[63,38353,38355],{"className":38354},[358],[63,38356,38358,38725],{"className":38357},[411,412],[63,38359,38361,38722],{"className":38360},[416],[63,38362,38364,38555,38563],{"className":38363,"style":37339},[420],[63,38365,38366,38369],{"style":15964},[63,38367],{"className":38368,"style":429},[428],[63,38370,38372,38444,38447,38498,38501,38552],{"className":38371},[399],[63,38373,38375,38378],{"className":38374},[399],[63,38376,5685],{"className":38377,"style":3251},[399,472],[63,38379,38381],{"className":38380},[476],[63,38382,38384,38436],{"className":38383},[411,412],[63,38385,38387,38433],{"className":38386},[416],[63,38388,38390],{"className":38389,"style":1532},[420],[63,38391,38392,38395],{"style":7483},[63,38393],{"className":38394,"style":817},[428],[63,38396,38398],{"className":38397},[433,434,435,436],[63,38399,38401],{"className":38400},[399,436],[63,38402,38404,38407],{"className":38403},[399,436],[63,38405,14101],{"className":38406,"style":3276},[399,472,436],[63,38408,38410],{"className":38409},[476],[63,38411,38413],{"className":38412},[411],[63,38414,38416],{"className":38415},[416],[63,38417,38419],{"className":38418,"style":19291},[420],[63,38420,38421,38424],{"style":19294},[63,38422],{"className":38423,"style":493},[428],[63,38425,38427],{"className":38426},[433,497,498,436],[63,38428,38430],{"className":38429},[399,436],[63,38431,19131],{"className":38432},[399,436],[63,38434,507],{"className":38435},[506],[63,38437,38439],{"className":38438},[416],[63,38440,38442],{"className":38441,"style":1554},[420],[63,38443],{},[63,38445,1078],{"className":38446},[403],[63,38448,38450,38453],{"className":38449},[399],[63,38451,38],{"className":38452},[399,472],[63,38454,38456],{"className":38455},[476],[63,38457,38459,38490],{"className":38458},[411,412],[63,38460,38462,38487],{"className":38461},[416],[63,38463,38465,38476],{"className":38464,"style":37441},[420],[63,38466,38467,38470],{"style":17457},[63,38468],{"className":38469,"style":817},[428],[63,38471,38473],{"className":38472},[433,434,435,436],[63,38474,612],{"className":38475},[399,472,436],[63,38477,38478,38481],{"style":16675},[63,38479],{"className":38480,"style":817},[428],[63,38482,38484],{"className":38483},[433,434,435,436],[63,38485,1057],{"className":38486},[399,472,436],[63,38488,507],{"className":38489},[506],[63,38491,38493],{"className":38492},[416],[63,38494,38496],{"className":38495,"style":17487},[420],[63,38497],{},[63,38499,13030],{"className":38500},[399],[63,38502,38504,38507],{"className":38503},[399],[63,38505,12890],{"className":38506},[399,472],[63,38508,38510],{"className":38509},[476],[63,38511,38513,38544],{"className":38512},[411,412],[63,38514,38516,38541],{"className":38515},[416],[63,38517,38519,38530],{"className":38518,"style":37441},[420],[63,38520,38521,38524],{"style":17457},[63,38522],{"className":38523,"style":817},[428],[63,38525,38527],{"className":38526},[433,434,435,436],[63,38528,612],{"className":38529},[399,472,436],[63,38531,38532,38535],{"style":16675},[63,38533],{"className":38534,"style":817},[428],[63,38536,38538],{"className":38537},[433,434,435,436],[63,38539,1057],{"className":38540},[399,472,436],[63,38542,507],{"className":38543},[506],[63,38545,38547],{"className":38546},[416],[63,38548,38550],{"className":38549,"style":17487},[420],[63,38551],{},[63,38553,1092],{"className":38554},[593],[63,38556,38557,38560],{"style":554},[63,38558],{"className":38559,"style":429},[428],[63,38561],{"className":38562,"style":562},[561],[63,38564,38565,38568],{"style":16030},[63,38566],{"className":38567,"style":429},[428],[63,38569,38571,38611,38614,38665,38668,38719],{"className":38570},[399],[63,38572,38574,38577],{"className":38573},[399],[63,38575,5685],{"className":38576,"style":3251},[399,472],[63,38578,38580],{"className":38579},[476],[63,38581,38583,38603],{"className":38582},[411,412],[63,38584,38586,38600],{"className":38585},[416],[63,38587,38589],{"className":38588,"style":1532},[420],[63,38590,38591,38594],{"style":7483},[63,38592],{"className":38593,"style":817},[428],[63,38595,38597],{"className":38596},[433,434,435,436],[63,38598,14101],{"className":38599,"style":3276},[399,472,436],[63,38601,507],{"className":38602},[506],[63,38604,38606],{"className":38605},[416],[63,38607,38609],{"className":38608,"style":1554},[420],[63,38610],{},[63,38612,1078],{"className":38613},[403],[63,38615,38617,38620],{"className":38616},[399],[63,38618,38],{"className":38619},[399,472],[63,38621,38623],{"className":38622},[476],[63,38624,38626,38657],{"className":38625},[411,412],[63,38627,38629,38654],{"className":38628},[416],[63,38630,38632,38643],{"className":38631,"style":18996},[420],[63,38633,38634,38637],{"style":17457},[63,38635],{"className":38636,"style":817},[428],[63,38638,38640],{"className":38639},[433,434,435,436],[63,38641,612],{"className":38642},[399,472,436],[63,38644,38645,38648],{"style":813},[63,38646],{"className":38647,"style":817},[428],[63,38649,38651],{"className":38650},[433,434,435,436],[63,38652,1057],{"className":38653},[399,472,436],[63,38655,507],{"className":38656},[506],[63,38658,38660],{"className":38659},[416],[63,38661,38663],{"className":38662,"style":17487},[420],[63,38664],{},[63,38666,13030],{"className":38667},[399],[63,38669,38671,38674],{"className":38670},[399],[63,38672,12890],{"className":38673},[399,472],[63,38675,38677],{"className":38676},[476],[63,38678,38680,38711],{"className":38679},[411,412],[63,38681,38683,38708],{"className":38682},[416],[63,38684,38686,38697],{"className":38685,"style":18996},[420],[63,38687,38688,38691],{"style":17457},[63,38689],{"className":38690,"style":817},[428],[63,38692,38694],{"className":38693},[433,434,435,436],[63,38695,612],{"className":38696},[399,472,436],[63,38698,38699,38702],{"style":813},[63,38700],{"className":38701,"style":817},[428],[63,38703,38705],{"className":38704},[433,434,435,436],[63,38706,1057],{"className":38707},[399,472,436],[63,38709,507],{"className":38710},[506],[63,38712,38714],{"className":38713},[416],[63,38715,38717],{"className":38716,"style":17487},[420],[63,38718],{},[63,38720,1092],{"className":38721},[593],[63,38723,507],{"className":38724},[506],[63,38726,38728],{"className":38727},[416],[63,38729,38731],{"className":38730,"style":16095},[420],[63,38732],{},[63,38734],{"className":38735},[593,404],[63,38737,38739,38742,39094],{"className":38738},[399],[63,38740],{"className":38741},[403,404],[63,38743,38745],{"className":38744},[358],[63,38746,38748,39086],{"className":38747},[411,412],[63,38749,38751,39083],{"className":38750},[416],[63,38752,38754,38913,38921],{"className":38753,"style":37339},[420],[63,38755,38756,38759],{"style":15964},[63,38757],{"className":38758,"style":429},[428],[63,38760,38762,38802,38805,38856,38859,38910],{"className":38761},[399],[63,38763,38765,38768],{"className":38764},[399],[63,38766,5685],{"className":38767,"style":3251},[399,472],[63,38769,38771],{"className":38770},[476],[63,38772,38774,38794],{"className":38773},[411,412],[63,38775,38777,38791],{"className":38776},[416],[63,38778,38780],{"className":38779,"style":1532},[420],[63,38781,38782,38785],{"style":7483},[63,38783],{"className":38784,"style":817},[428],[63,38786,38788],{"className":38787},[433,434,435,436],[63,38789,14101],{"className":38790,"style":3276},[399,472,436],[63,38792,507],{"className":38793},[506],[63,38795,38797],{"className":38796},[416],[63,38798,38800],{"className":38799,"style":1554},[420],[63,38801],{},[63,38803,1078],{"className":38804},[403],[63,38806,38808,38811],{"className":38807},[399],[63,38809,38],{"className":38810},[399,472],[63,38812,38814],{"className":38813},[476],[63,38815,38817,38848],{"className":38816},[411,412],[63,38818,38820,38845],{"className":38819},[416],[63,38821,38823,38834],{"className":38822,"style":37441},[420],[63,38824,38825,38828],{"style":17457},[63,38826],{"className":38827,"style":817},[428],[63,38829,38831],{"className":38830},[433,434,435,436],[63,38832,612],{"className":38833},[399,472,436],[63,38835,38836,38839],{"style":16675},[63,38837],{"className":38838,"style":817},[428],[63,38840,38842],{"className":38841},[433,434,435,436],[63,38843,1057],{"className":38844},[399,472,436],[63,38846,507],{"className":38847},[506],[63,38849,38851],{"className":38850},[416],[63,38852,38854],{"className":38853,"style":17487},[420],[63,38855],{},[63,38857,13030],{"className":38858},[399],[63,38860,38862,38865],{"className":38861},[399],[63,38863,12890],{"className":38864},[399,472],[63,38866,38868],{"className":38867},[476],[63,38869,38871,38902],{"className":38870},[411,412],[63,38872,38874,38899],{"className":38873},[416],[63,38875,38877,38888],{"className":38876,"style":37441},[420],[63,38878,38879,38882],{"style":17457},[63,38880],{"className":38881,"style":817},[428],[63,38883,38885],{"className":38884},[433,434,435,436],[63,38886,612],{"className":38887},[399,472,436],[63,38889,38890,38893],{"style":16675},[63,38891],{"className":38892,"style":817},[428],[63,38894,38896],{"className":38895},[433,434,435,436],[63,38897,1057],{"className":38898},[399,472,436],[63,38900,507],{"className":38901},[506],[63,38903,38905],{"className":38904},[416],[63,38906,38908],{"className":38907,"style":17487},[420],[63,38909],{},[63,38911,1092],{"className":38912},[593],[63,38914,38915,38918],{"style":554},[63,38916],{"className":38917,"style":429},[428],[63,38919],{"className":38920,"style":562},[561],[63,38922,38923,38926],{"style":16030},[63,38924],{"className":38925,"style":429},[428],[63,38927,38929,38932,38972,38975,39026,39029,39080],{"className":38928},[399],[63,38930,14525],{"className":38931},[399],[63,38933,38935,38938],{"className":38934},[399],[63,38936,5685],{"className":38937,"style":3251},[399,472],[63,38939,38941],{"className":38940},[476],[63,38942,38944,38964],{"className":38943},[411,412],[63,38945,38947,38961],{"className":38946},[416],[63,38948,38950],{"className":38949,"style":1532},[420],[63,38951,38952,38955],{"style":7483},[63,38953],{"className":38954,"style":817},[428],[63,38956,38958],{"className":38957},[433,434,435,436],[63,38959,14101],{"className":38960,"style":3276},[399,472,436],[63,38962,507],{"className":38963},[506],[63,38965,38967],{"className":38966},[416],[63,38968,38970],{"className":38969,"style":1554},[420],[63,38971],{},[63,38973,1078],{"className":38974},[403],[63,38976,38978,38981],{"className":38977},[399],[63,38979,38],{"className":38980},[399,472],[63,38982,38984],{"className":38983},[476],[63,38985,38987,39018],{"className":38986},[411,412],[63,38988,38990,39015],{"className":38989},[416],[63,38991,38993,39004],{"className":38992,"style":18996},[420],[63,38994,38995,38998],{"style":17457},[63,38996],{"className":38997,"style":817},[428],[63,38999,39001],{"className":39000},[433,434,435,436],[63,39002,612],{"className":39003},[399,472,436],[63,39005,39006,39009],{"style":813},[63,39007],{"className":39008,"style":817},[428],[63,39010,39012],{"className":39011},[433,434,435,436],[63,39013,1057],{"className":39014},[399,472,436],[63,39016,507],{"className":39017},[506],[63,39019,39021],{"className":39020},[416],[63,39022,39024],{"className":39023,"style":17487},[420],[63,39025],{},[63,39027,13030],{"className":39028},[399],[63,39030,39032,39035],{"className":39031},[399],[63,39033,12890],{"className":39034},[399,472],[63,39036,39038],{"className":39037},[476],[63,39039,39041,39072],{"className":39040},[411,412],[63,39042,39044,39069],{"className":39043},[416],[63,39045,39047,39058],{"className":39046,"style":18996},[420],[63,39048,39049,39052],{"style":17457},[63,39050],{"className":39051,"style":817},[428],[63,39053,39055],{"className":39054},[433,434,435,436],[63,39056,612],{"className":39057},[399,472,436],[63,39059,39060,39063],{"style":813},[63,39061],{"className":39062,"style":817},[428],[63,39064,39066],{"className":39065},[433,434,435,436],[63,39067,1057],{"className":39068},[399,472,436],[63,39070,507],{"className":39071},[506],[63,39073,39075],{"className":39074},[416],[63,39076,39078],{"className":39077,"style":17487},[420],[63,39079],{},[63,39081,1092],{"className":39082},[593],[63,39084,507],{"className":39085},[506],[63,39087,39089],{"className":39088},[416],[63,39090,39092],{"className":39091,"style":16095},[420],[63,39093],{},[63,39095],{"className":39096},[593,404],[63,39098,39099,39102],{"style":36230},[63,39100],{"className":39101,"style":15315},[428],[63,39103,39105,39108,39111,39114,39117,39179,39182,39249,39252,39356,39359,39451,39454,39505,39508,39511,39562,39565],{"className":39104},[399],[63,39106],{"className":39107},[399],[63,39109],{"className":39110,"style":777},[776],[63,39112,204],{"className":39113},[781],[63,39115],{"className":39116,"style":777},[776],[63,39118,39120,39123,39176],{"className":39119},[399],[63,39121],{"className":39122},[403,404],[63,39124,39126],{"className":39125},[358],[63,39127,39129,39168],{"className":39128},[411,412],[63,39130,39132,39165],{"className":39131},[416],[63,39133,39135,39146,39154],{"className":39134,"style":16424},[420],[63,39136,39137,39140],{"style":15964},[63,39138],{"className":39139,"style":429},[428],[63,39141,39143],{"className":39142},[399],[63,39144,7342],{"className":39145,"style":16436},[399,472],[63,39147,39148,39151],{"style":554},[63,39149],{"className":39150,"style":429},[428],[63,39152],{"className":39153,"style":562},[561],[63,39155,39156,39159],{"style":16030},[63,39157],{"className":39158,"style":429},[428],[63,39160,39162],{"className":39161},[399],[63,39163,363],{"className":39164},[399],[63,39166,507],{"className":39167},[506],[63,39169,39171],{"className":39170},[416],[63,39172,39174],{"className":39173,"style":16465},[420],[63,39175],{},[63,39177],{"className":39178},[593,404],[63,39180],{"className":39181,"style":1177},[776],[63,39183,39185],{"className":39184},[1212,14383],[63,39186,39188,39241],{"className":39187},[411,412],[63,39189,39191,39238],{"className":39190},[416],[63,39192,39194,39214,39224],{"className":39193,"style":16486},[420],[63,39195,39196,39199],{"style":16489},[63,39197],{"className":39198,"style":15577},[428],[63,39200,39202],{"className":39201},[433,434,435,436],[63,39203,39205,39208,39211],{"className":39204},[399,436],[63,39206,612],{"className":39207},[399,472,436],[63,39209,204],{"className":39210},[781,436],[63,39212,363],{"className":39213},[399,436],[63,39215,39216,39219],{"style":15586},[63,39217],{"className":39218,"style":15577},[428],[63,39220,39221],{},[63,39222,1874],{"className":39223},[1212,1958,15595],[63,39225,39226,39229],{"style":16520},[63,39227],{"className":39228,"style":15577},[428],[63,39230,39232],{"className":39231},[433,434,435,436],[63,39233,39235],{"className":39234},[399,436],[63,39236,7342],{"className":39237,"style":16436},[399,472,436],[63,39239,507],{"className":39240},[506],[63,39242,39244],{"className":39243},[416],[63,39245,39247],{"className":39246,"style":16539},[420],[63,39248],{},[63,39250],{"className":39251,"style":1177},[776],[63,39253,39255],{"className":39254},[1212,14383],[63,39256,39258,39348],{"className":39257},[411,412],[63,39259,39261,39345],{"className":39260},[416],[63,39262,39264,39284,39294],{"className":39263,"style":17295},[420],[63,39265,39266,39269],{"style":16489},[63,39267],{"className":39268,"style":15577},[428],[63,39270,39272],{"className":39271},[433,434,435,436],[63,39273,39275,39278,39281],{"className":39274},[399,436],[63,39276,1057],{"className":39277},[399,472,436],[63,39279,204],{"className":39280},[781,436],[63,39282,363],{"className":39283},[399,436],[63,39285,39286,39289],{"style":15586},[63,39287],{"className":39288,"style":15577},[428],[63,39290,39291],{},[63,39292,1874],{"className":39293},[1212,1958,15595],[63,39295,39296,39299],{"style":17328},[63,39297],{"className":39298,"style":15577},[428],[63,39300,39302],{"className":39301},[433,434,435,436],[63,39303,39305],{"className":39304},[399,436],[63,39306,39308,39311],{"className":39307},[399,436],[63,39309,13358],{"className":39310,"style":3251},[399,472,436],[63,39312,39314],{"className":39313},[476],[63,39315,39317,39337],{"className":39316},[411,412],[63,39318,39320,39334],{"className":39319},[416],[63,39321,39323],{"className":39322,"style":4367},[420],[63,39324,39325,39328],{"style":17358},[63,39326],{"className":39327,"style":493},[428],[63,39329,39331],{"className":39330},[433,497,498,436],[63,39332,612],{"className":39333},[399,472,436],[63,39335,507],{"className":39336},[506],[63,39338,39340],{"className":39339},[416],[63,39341,39343],{"className":39342,"style":4389},[420],[63,39344],{},[63,39346,507],{"className":39347},[506],[63,39349,39351],{"className":39350},[416],[63,39352,39354],{"className":39353,"style":16539},[420],[63,39355],{},[63,39357],{"className":39358,"style":1177},[776],[63,39360,39362,39365],{"className":39361},[399],[63,39363,1054],{"className":39364},[399,472],[63,39366,39368],{"className":39367},[476],[63,39369,39371,39443],{"className":39370},[411,412],[63,39372,39374,39440],{"className":39373},[416],[63,39375,39377,39420],{"className":39376,"style":25181},[420],[63,39378,39379,39382],{"style":17457},[63,39380],{"className":39381,"style":817},[428],[63,39383,39385],{"className":39384},[433,434,435,436],[63,39386,39388],{"className":39387},[399,436],[63,39389,39391,39394],{"className":39390},[399,436],[63,39392,14101],{"className":39393,"style":3276},[399,472,436],[63,39395,39397],{"className":39396},[476],[63,39398,39400],{"className":39399},[411],[63,39401,39403],{"className":39402},[416],[63,39404,39406],{"className":39405,"style":19291},[420],[63,39407,39408,39411],{"style":19294},[63,39409],{"className":39410,"style":493},[428],[63,39412,39414],{"className":39413},[433,497,498,436],[63,39415,39417],{"className":39416},[399,436],[63,39418,19131],{"className":39419},[399,436],[63,39421,39422,39425],{"style":16572},[63,39423],{"className":39424,"style":817},[428],[63,39426,39428],{"className":39427},[433,434,435,436],[63,39429,39431,39434,39437],{"className":39430},[399,436],[63,39432,27970],{"className":39433},[399,472,436],[63,39435,1054],{"className":39436},[399,472,436],[63,39438,5688],{"className":39439,"style":6101},[399,472,436],[63,39441,507],{"className":39442},[506],[63,39444,39446],{"className":39445},[416],[63,39447,39449],{"className":39448,"style":17487},[420],[63,39450],{},[63,39452,1078],{"className":39453},[403],[63,39455,39457,39460],{"className":39456},[399],[63,39458,12890],{"className":39459},[399,472],[63,39461,39463],{"className":39462},[476],[63,39464,39466,39497],{"className":39465},[411,412],[63,39467,39469,39494],{"className":39468},[416],[63,39470,39472,39483],{"className":39471,"style":17454},[420],[63,39473,39474,39477],{"style":17457},[63,39475],{"className":39476,"style":817},[428],[63,39478,39480],{"className":39479},[433,434,435,436],[63,39481,612],{"className":39482},[399,472,436],[63,39484,39485,39488],{"style":16572},[63,39486],{"className":39487,"style":817},[428],[63,39489,39491],{"className":39490},[433,434,435,436],[63,39492,1057],{"className":39493},[399,472,436],[63,39495,507],{"className":39496},[506],[63,39498,39500],{"className":39499},[416],[63,39501,39503],{"className":39502,"style":17487},[420],[63,39504],{},[63,39506,1083],{"className":39507},[1173],[63,39509],{"className":39510,"style":1177},[776],[63,39512,39514,39517],{"className":39513},[399],[63,39515,38],{"className":39516},[399,472],[63,39518,39520],{"className":39519},[476],[63,39521,39523,39554],{"className":39522},[411,412],[63,39524,39526,39551],{"className":39525},[416],[63,39527,39529,39540],{"className":39528,"style":17454},[420],[63,39530,39531,39534],{"style":17457},[63,39532],{"className":39533,"style":817},[428],[63,39535,39537],{"className":39536},[433,434,435,436],[63,39538,612],{"className":39539},[399,472,436],[63,39541,39542,39545],{"style":16572},[63,39543],{"className":39544,"style":817},[428],[63,39546,39548],{"className":39547},[433,434,435,436],[63,39549,1057],{"className":39550},[399,472,436],[63,39552,507],{"className":39553},[506],[63,39555,39557],{"className":39556},[416],[63,39558,39560],{"className":39559,"style":17487},[420],[63,39561],{},[63,39563,1092],{"className":39564},[593],[63,39566,39568,39571,39955],{"className":39567},[399],[63,39569],{"className":39570},[403,404],[63,39572,39574],{"className":39573},[358],[63,39575,39577,39947],{"className":39576},[411,412],[63,39578,39580,39944],{"className":39579},[416],[63,39581,39583,39774,39782],{"className":39582,"style":37339},[420],[63,39584,39585,39588],{"style":15964},[63,39586],{"className":39587,"style":429},[428],[63,39589,39591,39663,39666,39717,39720,39771],{"className":39590},[399],[63,39592,39594,39597],{"className":39593},[399],[63,39595,5685],{"className":39596,"style":3251},[399,472],[63,39598,39600],{"className":39599},[476],[63,39601,39603,39655],{"className":39602},[411,412],[63,39604,39606,39652],{"className":39605},[416],[63,39607,39609],{"className":39608,"style":1532},[420],[63,39610,39611,39614],{"style":7483},[63,39612],{"className":39613,"style":817},[428],[63,39615,39617],{"className":39616},[433,434,435,436],[63,39618,39620],{"className":39619},[399,436],[63,39621,39623,39626],{"className":39622},[399,436],[63,39624,14101],{"className":39625,"style":3276},[399,472,436],[63,39627,39629],{"className":39628},[476],[63,39630,39632],{"className":39631},[411],[63,39633,39635],{"className":39634},[416],[63,39636,39638],{"className":39637,"style":19291},[420],[63,39639,39640,39643],{"style":19294},[63,39641],{"className":39642,"style":493},[428],[63,39644,39646],{"className":39645},[433,497,498,436],[63,39647,39649],{"className":39648},[399,436],[63,39650,19131],{"className":39651},[399,436],[63,39653,507],{"className":39654},[506],[63,39656,39658],{"className":39657},[416],[63,39659,39661],{"className":39660,"style":1554},[420],[63,39662],{},[63,39664,1078],{"className":39665},[403],[63,39667,39669,39672],{"className":39668},[399],[63,39670,38],{"className":39671},[399,472],[63,39673,39675],{"className":39674},[476],[63,39676,39678,39709],{"className":39677},[411,412],[63,39679,39681,39706],{"className":39680},[416],[63,39682,39684,39695],{"className":39683,"style":37441},[420],[63,39685,39686,39689],{"style":17457},[63,39687],{"className":39688,"style":817},[428],[63,39690,39692],{"className":39691},[433,434,435,436],[63,39693,612],{"className":39694},[399,472,436],[63,39696,39697,39700],{"style":16675},[63,39698],{"className":39699,"style":817},[428],[63,39701,39703],{"className":39702},[433,434,435,436],[63,39704,1057],{"className":39705},[399,472,436],[63,39707,507],{"className":39708},[506],[63,39710,39712],{"className":39711},[416],[63,39713,39715],{"className":39714,"style":17487},[420],[63,39716],{},[63,39718,13030],{"className":39719},[399],[63,39721,39723,39726],{"className":39722},[399],[63,39724,12890],{"className":39725},[399,472],[63,39727,39729],{"className":39728},[476],[63,39730,39732,39763],{"className":39731},[411,412],[63,39733,39735,39760],{"className":39734},[416],[63,39736,39738,39749],{"className":39737,"style":37441},[420],[63,39739,39740,39743],{"style":17457},[63,39741],{"className":39742,"style":817},[428],[63,39744,39746],{"className":39745},[433,434,435,436],[63,39747,612],{"className":39748},[399,472,436],[63,39750,39751,39754],{"style":16675},[63,39752],{"className":39753,"style":817},[428],[63,39755,39757],{"className":39756},[433,434,435,436],[63,39758,1057],{"className":39759},[399,472,436],[63,39761,507],{"className":39762},[506],[63,39764,39766],{"className":39765},[416],[63,39767,39769],{"className":39768,"style":17487},[420],[63,39770],{},[63,39772,1092],{"className":39773},[593],[63,39775,39776,39779],{"style":554},[63,39777],{"className":39778,"style":429},[428],[63,39780],{"className":39781,"style":562},[561],[63,39783,39784,39787],{"style":16030},[63,39785],{"className":39786,"style":429},[428],[63,39788,39790,39793,39833,39836,39887,39890,39941],{"className":39789},[399],[63,39791,14525],{"className":39792},[399],[63,39794,39796,39799],{"className":39795},[399],[63,39797,5685],{"className":39798,"style":3251},[399,472],[63,39800,39802],{"className":39801},[476],[63,39803,39805,39825],{"className":39804},[411,412],[63,39806,39808,39822],{"className":39807},[416],[63,39809,39811],{"className":39810,"style":1532},[420],[63,39812,39813,39816],{"style":7483},[63,39814],{"className":39815,"style":817},[428],[63,39817,39819],{"className":39818},[433,434,435,436],[63,39820,14101],{"className":39821,"style":3276},[399,472,436],[63,39823,507],{"className":39824},[506],[63,39826,39828],{"className":39827},[416],[63,39829,39831],{"className":39830,"style":1554},[420],[63,39832],{},[63,39834,1078],{"className":39835},[403],[63,39837,39839,39842],{"className":39838},[399],[63,39840,38],{"className":39841},[399,472],[63,39843,39845],{"className":39844},[476],[63,39846,39848,39879],{"className":39847},[411,412],[63,39849,39851,39876],{"className":39850},[416],[63,39852,39854,39865],{"className":39853,"style":18996},[420],[63,39855,39856,39859],{"style":17457},[63,39857],{"className":39858,"style":817},[428],[63,39860,39862],{"className":39861},[433,434,435,436],[63,39863,612],{"className":39864},[399,472,436],[63,39866,39867,39870],{"style":813},[63,39868],{"className":39869,"style":817},[428],[63,39871,39873],{"className":39872},[433,434,435,436],[63,39874,1057],{"className":39875},[399,472,436],[63,39877,507],{"className":39878},[506],[63,39880,39882],{"className":39881},[416],[63,39883,39885],{"className":39884,"style":17487},[420],[63,39886],{},[63,39888,13030],{"className":39889},[399],[63,39891,39893,39896],{"className":39892},[399],[63,39894,12890],{"className":39895},[399,472],[63,39897,39899],{"className":39898},[476],[63,39900,39902,39933],{"className":39901},[411,412],[63,39903,39905,39930],{"className":39904},[416],[63,39906,39908,39919],{"className":39907,"style":18996},[420],[63,39909,39910,39913],{"style":17457},[63,39911],{"className":39912,"style":817},[428],[63,39914,39916],{"className":39915},[433,434,435,436],[63,39917,612],{"className":39918},[399,472,436],[63,39920,39921,39924],{"style":813},[63,39922],{"className":39923,"style":817},[428],[63,39925,39927],{"className":39926},[433,434,435,436],[63,39928,1057],{"className":39929},[399,472,436],[63,39931,507],{"className":39932},[506],[63,39934,39936],{"className":39935},[416],[63,39937,39939],{"className":39938,"style":17487},[420],[63,39940],{},[63,39942,1092],{"className":39943},[593],[63,39945,507],{"className":39946},[506],[63,39948,39950],{"className":39949},[416],[63,39951,39953],{"className":39952,"style":16095},[420],[63,39954],{},[63,39956],{"className":39957},[593,404],[63,39959,507],{"className":39960},[506],[63,39962,39964],{"className":39963},[416],[63,39965,39967],{"className":39966,"style":36246},[420],[63,39968],{},[14,39970,39971],{},"对应地，将期望最大化转换为损失最小化（取负号）：",[63,39973,39975],{"className":39974},[14495],[63,39976,39978,40146],{"className":39977},[342],[63,39979,39981],{"className":39980},[346],[348,39982,39983],{"xmlns":350,"display":9561},[352,39984,39985,40143],{},[355,39986,39987,39989,39991,39993,39995,39997,39999,40005,40019,40037,40055,40057,40065,40067,40075,40077],{},[371,39988,7842],{},[371,39990,1072],{},[371,39992,12890],{},[371,39994,12890],{},[738,39996,204],{},[738,39998,18511],{},[358,40000,40001,40003],{},[361,40002,363],{},[371,40004,7342],{},[14828,40006,40007,40009,40017],{},[738,40008,1874],{},[355,40010,40011,40013,40015],{},[371,40012,612],{},[738,40014,204],{},[361,40016,363],{},[371,40018,7342],{},[14828,40020,40021,40023,40031],{},[738,40022,1874],{},[355,40024,40025,40027,40029],{},[371,40026,1057],{},[738,40028,204],{},[361,40030,363],{},[368,40032,40033,40035],{},[371,40034,13358],{},[371,40036,612],{},[1870,40038,40039,40041,40047],{},[371,40040,1054],{},[742,40042,40043,40045],{},[371,40044,14101],{},[738,40046,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,40048,40049,40051,40053],{},[371,40050,27970],{},[371,40052,1054],{},[371,40054,5688],{},[738,40056,1078],{"stretchy":1077},[1870,40058,40059,40061,40063],{},[371,40060,12890],{},[371,40062,612],{},[371,40064,1057],{},[738,40066,1083],{"separator":386},[1870,40068,40069,40071,40073],{},[371,40070,38],{},[371,40072,612],{},[371,40074,1057],{},[738,40076,1092],{"stretchy":1077},[358,40078,40079,40109],{},[355,40080,40081,40087,40089,40097,40099,40107],{},[368,40082,40083,40085],{},[371,40084,5685],{},[371,40086,14101],{},[738,40088,1078],{"stretchy":1077},[1870,40090,40091,40093,40095],{},[371,40092,38],{},[371,40094,612],{},[371,40096,1057],{},[371,40098,13030],{"mathvariant":1053},[1870,40100,40101,40103,40105],{},[371,40102,12890],{},[371,40104,612],{},[371,40106,1057],{},[738,40108,1092],{"stretchy":1077},[355,40110,40111,40121,40123,40131,40133,40141],{},[368,40112,40113,40115],{},[371,40114,5685],{},[742,40116,40117,40119],{},[371,40118,14101],{},[738,40120,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,40122,1078],{"stretchy":1077},[1870,40124,40125,40127,40129],{},[371,40126,38],{},[371,40128,612],{},[371,40130,1057],{},[371,40132,13030],{"mathvariant":1053},[1870,40134,40135,40137,40139],{},[371,40136,12890],{},[371,40138,612],{},[371,40140,1057],{},[738,40142,1092],{"stretchy":1077},[378,40144,40145],{"encoding":380},"Loss = -\\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{P_\\theta(a_n^t | s_n^t)}{P_{\\theta'}(a_n^t | s_n^t)}",[63,40147,40149,40170],{"className":40148,"ariaHidden":386},[385],[63,40150,40152,40155,40158,40161,40164,40167],{"className":40151},[390],[63,40153],{"className":40154,"style":18606},[394],[63,40156,7842],{"className":40157},[399,472],[63,40159,18613],{"className":40160},[399,472],[63,40162],{"className":40163,"style":777},[776],[63,40165,204],{"className":40166},[781],[63,40168],{"className":40169,"style":777},[776],[63,40171,40173,40176,40179,40241,40244,40311,40314,40418,40421,40513,40516,40567,40570,40573,40624,40627],{"className":40172},[390],[63,40174],{"className":40175,"style":31737},[394],[63,40177,18511],{"className":40178},[399],[63,40180,40182,40185,40238],{"className":40181},[399],[63,40183],{"className":40184},[403,404],[63,40186,40188],{"className":40187},[358],[63,40189,40191,40230],{"className":40190},[411,412],[63,40192,40194,40227],{"className":40193},[416],[63,40195,40197,40208,40216],{"className":40196,"style":16424},[420],[63,40198,40199,40202],{"style":15964},[63,40200],{"className":40201,"style":429},[428],[63,40203,40205],{"className":40204},[399],[63,40206,7342],{"className":40207,"style":16436},[399,472],[63,40209,40210,40213],{"style":554},[63,40211],{"className":40212,"style":429},[428],[63,40214],{"className":40215,"style":562},[561],[63,40217,40218,40221],{"style":16030},[63,40219],{"className":40220,"style":429},[428],[63,40222,40224],{"className":40223},[399],[63,40225,363],{"className":40226},[399],[63,40228,507],{"className":40229},[506],[63,40231,40233],{"className":40232},[416],[63,40234,40236],{"className":40235,"style":16465},[420],[63,40237],{},[63,40239],{"className":40240},[593,404],[63,40242],{"className":40243,"style":1177},[776],[63,40245,40247],{"className":40246},[1212,14383],[63,40248,40250,40303],{"className":40249},[411,412],[63,40251,40253,40300],{"className":40252},[416],[63,40254,40256,40276,40286],{"className":40255,"style":16486},[420],[63,40257,40258,40261],{"style":16489},[63,40259],{"className":40260,"style":15577},[428],[63,40262,40264],{"className":40263},[433,434,435,436],[63,40265,40267,40270,40273],{"className":40266},[399,436],[63,40268,612],{"className":40269},[399,472,436],[63,40271,204],{"className":40272},[781,436],[63,40274,363],{"className":40275},[399,436],[63,40277,40278,40281],{"style":15586},[63,40279],{"className":40280,"style":15577},[428],[63,40282,40283],{},[63,40284,1874],{"className":40285},[1212,1958,15595],[63,40287,40288,40291],{"style":16520},[63,40289],{"className":40290,"style":15577},[428],[63,40292,40294],{"className":40293},[433,434,435,436],[63,40295,40297],{"className":40296},[399,436],[63,40298,7342],{"className":40299,"style":16436},[399,472,436],[63,40301,507],{"className":40302},[506],[63,40304,40306],{"className":40305},[416],[63,40307,40309],{"className":40308,"style":16539},[420],[63,40310],{},[63,40312],{"className":40313,"style":1177},[776],[63,40315,40317],{"className":40316},[1212,14383],[63,40318,40320,40410],{"className":40319},[411,412],[63,40321,40323,40407],{"className":40322},[416],[63,40324,40326,40346,40356],{"className":40325,"style":17295},[420],[63,40327,40328,40331],{"style":16489},[63,40329],{"className":40330,"style":15577},[428],[63,40332,40334],{"className":40333},[433,434,435,436],[63,40335,40337,40340,40343],{"className":40336},[399,436],[63,40338,1057],{"className":40339},[399,472,436],[63,40341,204],{"className":40342},[781,436],[63,40344,363],{"className":40345},[399,436],[63,40347,40348,40351],{"style":15586},[63,40349],{"className":40350,"style":15577},[428],[63,40352,40353],{},[63,40354,1874],{"className":40355},[1212,1958,15595],[63,40357,40358,40361],{"style":17328},[63,40359],{"className":40360,"style":15577},[428],[63,40362,40364],{"className":40363},[433,434,435,436],[63,40365,40367],{"className":40366},[399,436],[63,40368,40370,40373],{"className":40369},[399,436],[63,40371,13358],{"className":40372,"style":3251},[399,472,436],[63,40374,40376],{"className":40375},[476],[63,40377,40379,40399],{"className":40378},[411,412],[63,40380,40382,40396],{"className":40381},[416],[63,40383,40385],{"className":40384,"style":4367},[420],[63,40386,40387,40390],{"style":17358},[63,40388],{"className":40389,"style":493},[428],[63,40391,40393],{"className":40392},[433,497,498,436],[63,40394,612],{"className":40395},[399,472,436],[63,40397,507],{"className":40398},[506],[63,40400,40402],{"className":40401},[416],[63,40403,40405],{"className":40404,"style":4389},[420],[63,40406],{},[63,40408,507],{"className":40409},[506],[63,40411,40413],{"className":40412},[416],[63,40414,40416],{"className":40415,"style":16539},[420],[63,40417],{},[63,40419],{"className":40420,"style":1177},[776],[63,40422,40424,40427],{"className":40423},[399],[63,40425,1054],{"className":40426},[399,472],[63,40428,40430],{"className":40429},[476],[63,40431,40433,40505],{"className":40432},[411,412],[63,40434,40436,40502],{"className":40435},[416],[63,40437,40439,40482],{"className":40438,"style":25181},[420],[63,40440,40441,40444],{"style":17457},[63,40442],{"className":40443,"style":817},[428],[63,40445,40447],{"className":40446},[433,434,435,436],[63,40448,40450],{"className":40449},[399,436],[63,40451,40453,40456],{"className":40452},[399,436],[63,40454,14101],{"className":40455,"style":3276},[399,472,436],[63,40457,40459],{"className":40458},[476],[63,40460,40462],{"className":40461},[411],[63,40463,40465],{"className":40464},[416],[63,40466,40468],{"className":40467,"style":19291},[420],[63,40469,40470,40473],{"style":19294},[63,40471],{"className":40472,"style":493},[428],[63,40474,40476],{"className":40475},[433,497,498,436],[63,40477,40479],{"className":40478},[399,436],[63,40480,19131],{"className":40481},[399,436],[63,40483,40484,40487],{"style":16572},[63,40485],{"className":40486,"style":817},[428],[63,40488,40490],{"className":40489},[433,434,435,436],[63,40491,40493,40496,40499],{"className":40492},[399,436],[63,40494,27970],{"className":40495},[399,472,436],[63,40497,1054],{"className":40498},[399,472,436],[63,40500,5688],{"className":40501,"style":6101},[399,472,436],[63,40503,507],{"className":40504},[506],[63,40506,40508],{"className":40507},[416],[63,40509,40511],{"className":40510,"style":17487},[420],[63,40512],{},[63,40514,1078],{"className":40515},[403],[63,40517,40519,40522],{"className":40518},[399],[63,40520,12890],{"className":40521},[399,472],[63,40523,40525],{"className":40524},[476],[63,40526,40528,40559],{"className":40527},[411,412],[63,40529,40531,40556],{"className":40530},[416],[63,40532,40534,40545],{"className":40533,"style":17454},[420],[63,40535,40536,40539],{"style":17457},[63,40537],{"className":40538,"style":817},[428],[63,40540,40542],{"className":40541},[433,434,435,436],[63,40543,612],{"className":40544},[399,472,436],[63,40546,40547,40550],{"style":16572},[63,40548],{"className":40549,"style":817},[428],[63,40551,40553],{"className":40552},[433,434,435,436],[63,40554,1057],{"className":40555},[399,472,436],[63,40557,507],{"className":40558},[506],[63,40560,40562],{"className":40561},[416],[63,40563,40565],{"className":40564,"style":17487},[420],[63,40566],{},[63,40568,1083],{"className":40569},[1173],[63,40571],{"className":40572,"style":1177},[776],[63,40574,40576,40579],{"className":40575},[399],[63,40577,38],{"className":40578},[399,472],[63,40580,40582],{"className":40581},[476],[63,40583,40585,40616],{"className":40584},[411,412],[63,40586,40588,40613],{"className":40587},[416],[63,40589,40591,40602],{"className":40590,"style":17454},[420],[63,40592,40593,40596],{"style":17457},[63,40594],{"className":40595,"style":817},[428],[63,40597,40599],{"className":40598},[433,434,435,436],[63,40600,612],{"className":40601},[399,472,436],[63,40603,40604,40607],{"style":16572},[63,40605],{"className":40606,"style":817},[428],[63,40608,40610],{"className":40609},[433,434,435,436],[63,40611,1057],{"className":40612},[399,472,436],[63,40614,507],{"className":40615},[506],[63,40617,40619],{"className":40618},[416],[63,40620,40622],{"className":40621,"style":17487},[420],[63,40623],{},[63,40625,1092],{"className":40626},[593],[63,40628,40630,40633,41014],{"className":40629},[399],[63,40631],{"className":40632},[403,404],[63,40634,40636],{"className":40635},[358],[63,40637,40639,41006],{"className":40638},[411,412],[63,40640,40642,41003],{"className":40641},[416],[63,40643,40645,40836,40844],{"className":40644,"style":37339},[420],[63,40646,40647,40650],{"style":15964},[63,40648],{"className":40649,"style":429},[428],[63,40651,40653,40725,40728,40779,40782,40833],{"className":40652},[399],[63,40654,40656,40659],{"className":40655},[399],[63,40657,5685],{"className":40658,"style":3251},[399,472],[63,40660,40662],{"className":40661},[476],[63,40663,40665,40717],{"className":40664},[411,412],[63,40666,40668,40714],{"className":40667},[416],[63,40669,40671],{"className":40670,"style":1532},[420],[63,40672,40673,40676],{"style":7483},[63,40674],{"className":40675,"style":817},[428],[63,40677,40679],{"className":40678},[433,434,435,436],[63,40680,40682],{"className":40681},[399,436],[63,40683,40685,40688],{"className":40684},[399,436],[63,40686,14101],{"className":40687,"style":3276},[399,472,436],[63,40689,40691],{"className":40690},[476],[63,40692,40694],{"className":40693},[411],[63,40695,40697],{"className":40696},[416],[63,40698,40700],{"className":40699,"style":19291},[420],[63,40701,40702,40705],{"style":19294},[63,40703],{"className":40704,"style":493},[428],[63,40706,40708],{"className":40707},[433,497,498,436],[63,40709,40711],{"className":40710},[399,436],[63,40712,19131],{"className":40713},[399,436],[63,40715,507],{"className":40716},[506],[63,40718,40720],{"className":40719},[416],[63,40721,40723],{"className":40722,"style":1554},[420],[63,40724],{},[63,40726,1078],{"className":40727},[403],[63,40729,40731,40734],{"className":40730},[399],[63,40732,38],{"className":40733},[399,472],[63,40735,40737],{"className":40736},[476],[63,40738,40740,40771],{"className":40739},[411,412],[63,40741,40743,40768],{"className":40742},[416],[63,40744,40746,40757],{"className":40745,"style":37441},[420],[63,40747,40748,40751],{"style":17457},[63,40749],{"className":40750,"style":817},[428],[63,40752,40754],{"className":40753},[433,434,435,436],[63,40755,612],{"className":40756},[399,472,436],[63,40758,40759,40762],{"style":16675},[63,40760],{"className":40761,"style":817},[428],[63,40763,40765],{"className":40764},[433,434,435,436],[63,40766,1057],{"className":40767},[399,472,436],[63,40769,507],{"className":40770},[506],[63,40772,40774],{"className":40773},[416],[63,40775,40777],{"className":40776,"style":17487},[420],[63,40778],{},[63,40780,13030],{"className":40781},[399],[63,40783,40785,40788],{"className":40784},[399],[63,40786,12890],{"className":40787},[399,472],[63,40789,40791],{"className":40790},[476],[63,40792,40794,40825],{"className":40793},[411,412],[63,40795,40797,40822],{"className":40796},[416],[63,40798,40800,40811],{"className":40799,"style":37441},[420],[63,40801,40802,40805],{"style":17457},[63,40803],{"className":40804,"style":817},[428],[63,40806,40808],{"className":40807},[433,434,435,436],[63,40809,612],{"className":40810},[399,472,436],[63,40812,40813,40816],{"style":16675},[63,40814],{"className":40815,"style":817},[428],[63,40817,40819],{"className":40818},[433,434,435,436],[63,40820,1057],{"className":40821},[399,472,436],[63,40823,507],{"className":40824},[506],[63,40826,40828],{"className":40827},[416],[63,40829,40831],{"className":40830,"style":17487},[420],[63,40832],{},[63,40834,1092],{"className":40835},[593],[63,40837,40838,40841],{"style":554},[63,40839],{"className":40840,"style":429},[428],[63,40842],{"className":40843,"style":562},[561],[63,40845,40846,40849],{"style":16030},[63,40847],{"className":40848,"style":429},[428],[63,40850,40852,40892,40895,40946,40949,41000],{"className":40851},[399],[63,40853,40855,40858],{"className":40854},[399],[63,40856,5685],{"className":40857,"style":3251},[399,472],[63,40859,40861],{"className":40860},[476],[63,40862,40864,40884],{"className":40863},[411,412],[63,40865,40867,40881],{"className":40866},[416],[63,40868,40870],{"className":40869,"style":1532},[420],[63,40871,40872,40875],{"style":7483},[63,40873],{"className":40874,"style":817},[428],[63,40876,40878],{"className":40877},[433,434,435,436],[63,40879,14101],{"className":40880,"style":3276},[399,472,436],[63,40882,507],{"className":40883},[506],[63,40885,40887],{"className":40886},[416],[63,40888,40890],{"className":40889,"style":1554},[420],[63,40891],{},[63,40893,1078],{"className":40894},[403],[63,40896,40898,40901],{"className":40897},[399],[63,40899,38],{"className":40900},[399,472],[63,40902,40904],{"className":40903},[476],[63,40905,40907,40938],{"className":40906},[411,412],[63,40908,40910,40935],{"className":40909},[416],[63,40911,40913,40924],{"className":40912,"style":18996},[420],[63,40914,40915,40918],{"style":17457},[63,40916],{"className":40917,"style":817},[428],[63,40919,40921],{"className":40920},[433,434,435,436],[63,40922,612],{"className":40923},[399,472,436],[63,40925,40926,40929],{"style":813},[63,40927],{"className":40928,"style":817},[428],[63,40930,40932],{"className":40931},[433,434,435,436],[63,40933,1057],{"className":40934},[399,472,436],[63,40936,507],{"className":40937},[506],[63,40939,40941],{"className":40940},[416],[63,40942,40944],{"className":40943,"style":17487},[420],[63,40945],{},[63,40947,13030],{"className":40948},[399],[63,40950,40952,40955],{"className":40951},[399],[63,40953,12890],{"className":40954},[399,472],[63,40956,40958],{"className":40957},[476],[63,40959,40961,40992],{"className":40960},[411,412],[63,40962,40964,40989],{"className":40963},[416],[63,40965,40967,40978],{"className":40966,"style":18996},[420],[63,40968,40969,40972],{"style":17457},[63,40970],{"className":40971,"style":817},[428],[63,40973,40975],{"className":40974},[433,434,435,436],[63,40976,612],{"className":40977},[399,472,436],[63,40979,40980,40983],{"style":813},[63,40981],{"className":40982,"style":817},[428],[63,40984,40986],{"className":40985},[433,434,435,436],[63,40987,1057],{"className":40988},[399,472,436],[63,40990,507],{"className":40991},[506],[63,40993,40995],{"className":40994},[416],[63,40996,40998],{"className":40997,"style":17487},[420],[63,40999],{},[63,41001,1092],{"className":41002},[593],[63,41004,507],{"className":41005},[506],[63,41007,41009],{"className":41008},[416],[63,41010,41012],{"className":41011,"style":16095},[420],[63,41013],{},[63,41015],{"className":41016},[593,404],[14,41018,41019,41020,41023,41024,4629,41052,41113,41114,41337],{},"这里有一个 ",[11217,41021,41022],{},"隐含的前提","：重要性采样要求 ",[63,41025,41027,41040],{"className":41026},[342],[63,41028,41030],{"className":41029},[346],[348,41031,41032],{"xmlns":350},[352,41033,41034,41038],{},[355,41035,41036],{},[371,41037,14101],{},[378,41039,34687],{"encoding":380},[63,41041,41043],{"className":41042,"ariaHidden":386},[385],[63,41044,41046,41049],{"className":41045},[390],[63,41047],{"className":41048,"style":684},[394],[63,41050,14101],{"className":41051,"style":3276},[399,472],[63,41053,41055,41072],{"className":41054},[342],[63,41056,41058],{"className":41057},[346],[348,41059,41060],{"xmlns":350},[352,41061,41062,41070],{},[355,41063,41064],{},[742,41065,41066,41068],{},[371,41067,14101],{},[738,41069,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[378,41071,34624],{"encoding":380},[63,41073,41075],{"className":41074,"ariaHidden":386},[385],[63,41076,41078,41081],{"className":41077},[390],[63,41079],{"className":41080,"style":34634},[394],[63,41082,41084,41087],{"className":41083},[399],[63,41085,14101],{"className":41086,"style":3276},[399,472],[63,41088,41090],{"className":41089},[476],[63,41091,41093],{"className":41092},[411],[63,41094,41096],{"className":41095},[416],[63,41097,41099],{"className":41098,"style":34634},[420],[63,41100,41101,41104],{"style":813},[63,41102],{"className":41103,"style":817},[428],[63,41105,41107],{"className":41106},[433,434,435,436],[63,41108,41110],{"className":41109},[399,436],[63,41111,19131],{"className":41112},[399,436]," 不能差太多，否则比值 ",[63,41115,41117,41147],{"className":41116},[342],[63,41118,41120],{"className":41119},[346],[348,41121,41122],{"xmlns":350},[352,41123,41124,41144],{},[355,41125,41126],{},[358,41127,41128,41134],{},[368,41129,41130,41132],{},[371,41131,5685],{},[371,41133,14101],{},[368,41135,41136,41138],{},[371,41137,5685],{},[742,41139,41140,41142],{},[371,41141,14101],{},[738,41143,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[378,41145,41146],{"encoding":380},"\\frac{P_\\theta}{P_{\\theta'}}",[63,41148,41150],{"className":41149,"ariaHidden":386},[385],[63,41151,41153,41157],{"className":41152},[390],[63,41154],{"className":41155,"style":41156},[394],"height:1.4226em;vertical-align:-0.5284em;",[63,41158,41160,41163,41334],{"className":41159},[399],[63,41161],{"className":41162},[403,404],[63,41164,41166],{"className":41165},[358],[63,41167,41169,41325],{"className":41168},[411,412],[63,41170,41172,41322],{"className":41171},[416],[63,41173,41176,41262,41270],{"className":41174,"style":41175},[420],"height:0.8942em;",[63,41177,41178,41181],{"style":18653},[63,41179],{"className":41180,"style":429},[428],[63,41182,41184],{"className":41183},[433,434,435,436],[63,41185,41187],{"className":41186},[399,436],[63,41188,41190,41193],{"className":41189},[399,436],[63,41191,5685],{"className":41192,"style":3251},[399,472,436],[63,41194,41196],{"className":41195},[476],[63,41197,41199,41253],{"className":41198},[411,412],[63,41200,41202,41250],{"className":41201},[416],[63,41203,41205],{"className":41204,"style":486},[420],[63,41206,41208,41212],{"style":41207},"top:-2.3448em;margin-left:-0.1389em;margin-right:0.0714em;",[63,41209],{"className":41210,"style":41211},[428],"height:2.6068em;",[63,41213,41215],{"className":41214},[433,497,498,436],[63,41216,41218],{"className":41217},[399,436],[63,41219,41221,41224],{"className":41220},[399,436],[63,41222,14101],{"className":41223,"style":3276},[399,472,436],[63,41225,41227],{"className":41226},[476],[63,41228,41230],{"className":41229},[411],[63,41231,41233],{"className":41232},[416],[63,41234,41237],{"className":41235,"style":41236},[420],"height:0.8496em;",[63,41238,41240,41244],{"style":41239},"top:-2.8496em;margin-right:0.1em;",[63,41241],{"className":41242,"style":41243},[428],"height:2.5556em;",[63,41245,41247],{"className":41246},[399,436],[63,41248,19131],{"className":41249},[399,436],[63,41251,507],{"className":41252},[506],[63,41254,41256],{"className":41255},[416],[63,41257,41260],{"className":41258,"style":41259},[420],"height:0.262em;",[63,41261],{},[63,41263,41264,41267],{"style":554},[63,41265],{"className":41266,"style":429},[428],[63,41268],{"className":41269,"style":562},[561],[63,41271,41273,41276],{"style":41272},"top:-3.4159em;",[63,41274],{"className":41275,"style":429},[428],[63,41277,41279],{"className":41278},[433,434,435,436],[63,41280,41282],{"className":41281},[399,436],[63,41283,41285,41288],{"className":41284},[399,436],[63,41286,5685],{"className":41287,"style":3251},[399,472,436],[63,41289,41291],{"className":41290},[476],[63,41292,41294,41314],{"className":41293},[411,412],[63,41295,41297,41311],{"className":41296},[416],[63,41298,41300],{"className":41299,"style":486},[420],[63,41301,41302,41305],{"style":14320},[63,41303],{"className":41304,"style":493},[428],[63,41306,41308],{"className":41307},[433,497,498,436],[63,41309,14101],{"className":41310,"style":3276},[399,472,436],[63,41312,507],{"className":41313},[506],[63,41315,41317],{"className":41316},[416],[63,41318,41320],{"className":41319,"style":514},[420],[63,41321],{},[63,41323,507],{"className":41324},[506],[63,41326,41328],{"className":41327},[416],[63,41329,41332],{"className":41330,"style":41331},[420],"height:0.5284em;",[63,41333],{},[63,41335],{"className":41336},[593,404]," 的方差会爆炸，估计就会失真。换句话说，我们需要给\"新旧策略的差距\"施加一个约束。PPO 给出了两种约束方式：",[41339,41340,41342],"h4",{"id":41341},"ppo-penalty在-surrogate-上加-kl-惩罚","PPO-Penalty：在 surrogate 上加 KL 惩罚",[14,41344,41345,41346,41378,41379,41407],{},"如何让训练策略与参考策略不至于偏离太远？最直观的做法是给目标函数加一个 KL 散度惩罚项。KL 散度衡量两个分布的差异：差异越小，KL 越接近 0；差异越大，KL 越大。我们用一个权重 ",[63,41347,41349,41364],{"className":41348},[342],[63,41350,41352],{"className":41351},[346],[348,41353,41354],{"xmlns":350},[352,41355,41356,41361],{},[355,41357,41358],{},[371,41359,41360],{},"β",[378,41362,41363],{"encoding":380},"\\beta",[63,41365,41367],{"className":41366,"ariaHidden":386},[385],[63,41368,41370,41374],{"className":41369},[390],[63,41371],{"className":41372,"style":41373},[394],"height:0.8889em;vertical-align:-0.1944em;",[63,41375,41360],{"className":41376,"style":41377},[399,472],"margin-right:0.05278em;"," 来控制这一惩罚的强度（实际中 ",[63,41380,41382,41395],{"className":41381},[342],[63,41383,41385],{"className":41384},[346],[348,41386,41387],{"xmlns":350},[352,41388,41389,41393],{},[355,41390,41391],{},[371,41392,41360],{},[378,41394,41363],{"encoding":380},[63,41396,41398],{"className":41397,"ariaHidden":386},[385],[63,41399,41401,41404],{"className":41400},[390],[63,41402],{"className":41403,"style":41373},[394],[63,41405,41360],{"className":41406,"style":41377},[399,472]," 还会做自适应调整）：",[63,41409,41411],{"className":41410},[14495],[63,41412,41414,41621],{"className":41413},[342],[63,41415,41417],{"className":41416},[346],[348,41418,41419],{"xmlns":350,"display":9561},[352,41420,41421,41618],{},[355,41422,41423,41425,41427,41429,41436,41438,41440,41446,41460,41478,41496,41498,41506,41508,41516,41518,41584,41586,41588,41590,41596,41598,41604,41606,41616],{},[371,41424,7842],{},[371,41426,1072],{},[371,41428,12890],{},[368,41430,41431,41433],{},[371,41432,12890],{},[3343,41434,41435],{},"ppo",[738,41437,204],{},[738,41439,18511],{},[358,41441,41442,41444],{},[361,41443,363],{},[371,41445,7342],{},[14828,41447,41448,41450,41458],{},[738,41449,1874],{},[355,41451,41452,41454,41456],{},[371,41453,612],{},[738,41455,204],{},[361,41457,363],{},[371,41459,7342],{},[14828,41461,41462,41464,41472],{},[738,41463,1874],{},[355,41465,41466,41468,41470],{},[371,41467,1057],{},[738,41469,204],{},[361,41471,363],{},[368,41473,41474,41476],{},[371,41475,13358],{},[371,41477,612],{},[1870,41479,41480,41482,41488],{},[371,41481,1054],{},[742,41483,41484,41486],{},[371,41485,14101],{},[738,41487,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,41489,41490,41492,41494],{},[371,41491,27970],{},[371,41493,1054],{},[371,41495,5688],{},[738,41497,1078],{"stretchy":1077},[1870,41499,41500,41502,41504],{},[371,41501,12890],{},[371,41503,612],{},[371,41505,1057],{},[738,41507,1083],{"separator":386},[1870,41509,41510,41512,41514],{},[371,41511,38],{},[371,41513,612],{},[371,41515,1057],{},[738,41517,1092],{"stretchy":1077},[358,41519,41520,41550],{},[355,41521,41522,41528,41530,41538,41540,41548],{},[368,41523,41524,41526],{},[371,41525,5685],{},[371,41527,14101],{},[738,41529,1078],{"stretchy":1077},[1870,41531,41532,41534,41536],{},[371,41533,38],{},[371,41535,612],{},[371,41537,1057],{},[371,41539,13030],{"mathvariant":1053},[1870,41541,41542,41544,41546],{},[371,41543,12890],{},[371,41545,612],{},[371,41547,1057],{},[738,41549,1092],{"stretchy":1077},[355,41551,41552,41562,41564,41572,41574,41582],{},[368,41553,41554,41556],{},[371,41555,5685],{},[742,41557,41558,41560],{},[371,41559,14101],{},[738,41561,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,41563,1078],{"stretchy":1077},[1870,41565,41566,41568,41570],{},[371,41567,38],{},[371,41569,612],{},[371,41571,1057],{},[371,41573,13030],{"mathvariant":1053},[1870,41575,41576,41578,41580],{},[371,41577,12890],{},[371,41579,612],{},[371,41581,1057],{},[738,41583,1092],{"stretchy":1077},[738,41585,5839],{},[371,41587,41360],{},[3343,41589,7365],{},[355,41591,41592,41594],{},[371,41593,849],{"mathvariant":1053},[371,41595,7842],{"mathvariant":1053},[738,41597,1078],{"stretchy":1077},[368,41599,41600,41602],{},[371,41601,5685],{},[371,41603,14101],{},[738,41605,1083],{"separator":386},[368,41607,41608,41610],{},[371,41609,5685],{},[742,41611,41612,41614],{},[371,41613,14101],{},[738,41615,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,41617,1092],{"stretchy":1077},[378,41619,41620],{"encoding":380},"Loss_{\\text{ppo}} = -\\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{P_{\\theta}(a_n^t|s_n^t)}{P_{\\theta'}(a_n^t|s_n^t)} + \\beta\\,\\mathrm{KL}(P_{\\theta}, P_{\\theta'})",[63,41622,41624,41694,42553],{"className":41623,"ariaHidden":386},[385],[63,41625,41627,41631,41634,41638,41685,41688,41691],{"className":41626},[390],[63,41628],{"className":41629,"style":41630},[394],"height:0.9694em;vertical-align:-0.2861em;",[63,41632,7842],{"className":41633},[399,472],[63,41635,41637],{"className":41636},[399,472],"os",[63,41639,41641,41644],{"className":41640},[399],[63,41642,12890],{"className":41643},[399,472],[63,41645,41647],{"className":41646},[476],[63,41648,41650,41676],{"className":41649},[411,412],[63,41651,41653,41673],{"className":41652},[416],[63,41654,41656],{"className":41655,"style":8342},[420],[63,41657,41658,41661],{"style":1535},[63,41659],{"className":41660,"style":817},[428],[63,41662,41664],{"className":41663},[433,434,435,436],[63,41665,41667],{"className":41666},[399,436],[63,41668,41670],{"className":41669},[399,5395,436],[63,41671,41435],{"className":41672},[399,436],[63,41674,507],{"className":41675},[506],[63,41677,41679],{"className":41678},[416],[63,41680,41683],{"className":41681,"style":41682},[420],"height:0.2861em;",[63,41684],{},[63,41686],{"className":41687,"style":777},[776],[63,41689,204],{"className":41690},[781],[63,41692],{"className":41693,"style":777},[776],[63,41695,41697,41700,41703,41765,41768,41835,41838,41942,41945,42037,42040,42091,42094,42097,42148,42151,42544,42547,42550],{"className":41696},[390],[63,41698],{"className":41699,"style":31737},[394],[63,41701,18511],{"className":41702},[399],[63,41704,41706,41709,41762],{"className":41705},[399],[63,41707],{"className":41708},[403,404],[63,41710,41712],{"className":41711},[358],[63,41713,41715,41754],{"className":41714},[411,412],[63,41716,41718,41751],{"className":41717},[416],[63,41719,41721,41732,41740],{"className":41720,"style":16424},[420],[63,41722,41723,41726],{"style":15964},[63,41724],{"className":41725,"style":429},[428],[63,41727,41729],{"className":41728},[399],[63,41730,7342],{"className":41731,"style":16436},[399,472],[63,41733,41734,41737],{"style":554},[63,41735],{"className":41736,"style":429},[428],[63,41738],{"className":41739,"style":562},[561],[63,41741,41742,41745],{"style":16030},[63,41743],{"className":41744,"style":429},[428],[63,41746,41748],{"className":41747},[399],[63,41749,363],{"className":41750},[399],[63,41752,507],{"className":41753},[506],[63,41755,41757],{"className":41756},[416],[63,41758,41760],{"className":41759,"style":16465},[420],[63,41761],{},[63,41763],{"className":41764},[593,404],[63,41766],{"className":41767,"style":1177},[776],[63,41769,41771],{"className":41770},[1212,14383],[63,41772,41774,41827],{"className":41773},[411,412],[63,41775,41777,41824],{"className":41776},[416],[63,41778,41780,41800,41810],{"className":41779,"style":16486},[420],[63,41781,41782,41785],{"style":16489},[63,41783],{"className":41784,"style":15577},[428],[63,41786,41788],{"className":41787},[433,434,435,436],[63,41789,41791,41794,41797],{"className":41790},[399,436],[63,41792,612],{"className":41793},[399,472,436],[63,41795,204],{"className":41796},[781,436],[63,41798,363],{"className":41799},[399,436],[63,41801,41802,41805],{"style":15586},[63,41803],{"className":41804,"style":15577},[428],[63,41806,41807],{},[63,41808,1874],{"className":41809},[1212,1958,15595],[63,41811,41812,41815],{"style":16520},[63,41813],{"className":41814,"style":15577},[428],[63,41816,41818],{"className":41817},[433,434,435,436],[63,41819,41821],{"className":41820},[399,436],[63,41822,7342],{"className":41823,"style":16436},[399,472,436],[63,41825,507],{"className":41826},[506],[63,41828,41830],{"className":41829},[416],[63,41831,41833],{"className":41832,"style":16539},[420],[63,41834],{},[63,41836],{"className":41837,"style":1177},[776],[63,41839,41841],{"className":41840},[1212,14383],[63,41842,41844,41934],{"className":41843},[411,412],[63,41845,41847,41931],{"className":41846},[416],[63,41848,41850,41870,41880],{"className":41849,"style":17295},[420],[63,41851,41852,41855],{"style":16489},[63,41853],{"className":41854,"style":15577},[428],[63,41856,41858],{"className":41857},[433,434,435,436],[63,41859,41861,41864,41867],{"className":41860},[399,436],[63,41862,1057],{"className":41863},[399,472,436],[63,41865,204],{"className":41866},[781,436],[63,41868,363],{"className":41869},[399,436],[63,41871,41872,41875],{"style":15586},[63,41873],{"className":41874,"style":15577},[428],[63,41876,41877],{},[63,41878,1874],{"className":41879},[1212,1958,15595],[63,41881,41882,41885],{"style":17328},[63,41883],{"className":41884,"style":15577},[428],[63,41886,41888],{"className":41887},[433,434,435,436],[63,41889,41891],{"className":41890},[399,436],[63,41892,41894,41897],{"className":41893},[399,436],[63,41895,13358],{"className":41896,"style":3251},[399,472,436],[63,41898,41900],{"className":41899},[476],[63,41901,41903,41923],{"className":41902},[411,412],[63,41904,41906,41920],{"className":41905},[416],[63,41907,41909],{"className":41908,"style":4367},[420],[63,41910,41911,41914],{"style":17358},[63,41912],{"className":41913,"style":493},[428],[63,41915,41917],{"className":41916},[433,497,498,436],[63,41918,612],{"className":41919},[399,472,436],[63,41921,507],{"className":41922},[506],[63,41924,41926],{"className":41925},[416],[63,41927,41929],{"className":41928,"style":4389},[420],[63,41930],{},[63,41932,507],{"className":41933},[506],[63,41935,41937],{"className":41936},[416],[63,41938,41940],{"className":41939,"style":16539},[420],[63,41941],{},[63,41943],{"className":41944,"style":1177},[776],[63,41946,41948,41951],{"className":41947},[399],[63,41949,1054],{"className":41950},[399,472],[63,41952,41954],{"className":41953},[476],[63,41955,41957,42029],{"className":41956},[411,412],[63,41958,41960,42026],{"className":41959},[416],[63,41961,41963,42006],{"className":41962,"style":25181},[420],[63,41964,41965,41968],{"style":17457},[63,41966],{"className":41967,"style":817},[428],[63,41969,41971],{"className":41970},[433,434,435,436],[63,41972,41974],{"className":41973},[399,436],[63,41975,41977,41980],{"className":41976},[399,436],[63,41978,14101],{"className":41979,"style":3276},[399,472,436],[63,41981,41983],{"className":41982},[476],[63,41984,41986],{"className":41985},[411],[63,41987,41989],{"className":41988},[416],[63,41990,41992],{"className":41991,"style":19291},[420],[63,41993,41994,41997],{"style":19294},[63,41995],{"className":41996,"style":493},[428],[63,41998,42000],{"className":41999},[433,497,498,436],[63,42001,42003],{"className":42002},[399,436],[63,42004,19131],{"className":42005},[399,436],[63,42007,42008,42011],{"style":16572},[63,42009],{"className":42010,"style":817},[428],[63,42012,42014],{"className":42013},[433,434,435,436],[63,42015,42017,42020,42023],{"className":42016},[399,436],[63,42018,27970],{"className":42019},[399,472,436],[63,42021,1054],{"className":42022},[399,472,436],[63,42024,5688],{"className":42025,"style":6101},[399,472,436],[63,42027,507],{"className":42028},[506],[63,42030,42032],{"className":42031},[416],[63,42033,42035],{"className":42034,"style":17487},[420],[63,42036],{},[63,42038,1078],{"className":42039},[403],[63,42041,42043,42046],{"className":42042},[399],[63,42044,12890],{"className":42045},[399,472],[63,42047,42049],{"className":42048},[476],[63,42050,42052,42083],{"className":42051},[411,412],[63,42053,42055,42080],{"className":42054},[416],[63,42056,42058,42069],{"className":42057,"style":17454},[420],[63,42059,42060,42063],{"style":17457},[63,42061],{"className":42062,"style":817},[428],[63,42064,42066],{"className":42065},[433,434,435,436],[63,42067,612],{"className":42068},[399,472,436],[63,42070,42071,42074],{"style":16572},[63,42072],{"className":42073,"style":817},[428],[63,42075,42077],{"className":42076},[433,434,435,436],[63,42078,1057],{"className":42079},[399,472,436],[63,42081,507],{"className":42082},[506],[63,42084,42086],{"className":42085},[416],[63,42087,42089],{"className":42088,"style":17487},[420],[63,42090],{},[63,42092,1083],{"className":42093},[1173],[63,42095],{"className":42096,"style":1177},[776],[63,42098,42100,42103],{"className":42099},[399],[63,42101,38],{"className":42102},[399,472],[63,42104,42106],{"className":42105},[476],[63,42107,42109,42140],{"className":42108},[411,412],[63,42110,42112,42137],{"className":42111},[416],[63,42113,42115,42126],{"className":42114,"style":17454},[420],[63,42116,42117,42120],{"style":17457},[63,42118],{"className":42119,"style":817},[428],[63,42121,42123],{"className":42122},[433,434,435,436],[63,42124,612],{"className":42125},[399,472,436],[63,42127,42128,42131],{"style":16572},[63,42129],{"className":42130,"style":817},[428],[63,42132,42134],{"className":42133},[433,434,435,436],[63,42135,1057],{"className":42136},[399,472,436],[63,42138,507],{"className":42139},[506],[63,42141,42143],{"className":42142},[416],[63,42144,42146],{"className":42145,"style":17487},[420],[63,42147],{},[63,42149,1092],{"className":42150},[593],[63,42152,42154,42157,42541],{"className":42153},[399],[63,42155],{"className":42156},[403,404],[63,42158,42160],{"className":42159},[358],[63,42161,42163,42533],{"className":42162},[411,412],[63,42164,42166,42530],{"className":42165},[416],[63,42167,42169,42360,42368],{"className":42168,"style":37339},[420],[63,42170,42171,42174],{"style":15964},[63,42172],{"className":42173,"style":429},[428],[63,42175,42177,42249,42252,42303,42306,42357],{"className":42176},[399],[63,42178,42180,42183],{"className":42179},[399],[63,42181,5685],{"className":42182,"style":3251},[399,472],[63,42184,42186],{"className":42185},[476],[63,42187,42189,42241],{"className":42188},[411,412],[63,42190,42192,42238],{"className":42191},[416],[63,42193,42195],{"className":42194,"style":1532},[420],[63,42196,42197,42200],{"style":7483},[63,42198],{"className":42199,"style":817},[428],[63,42201,42203],{"className":42202},[433,434,435,436],[63,42204,42206],{"className":42205},[399,436],[63,42207,42209,42212],{"className":42208},[399,436],[63,42210,14101],{"className":42211,"style":3276},[399,472,436],[63,42213,42215],{"className":42214},[476],[63,42216,42218],{"className":42217},[411],[63,42219,42221],{"className":42220},[416],[63,42222,42224],{"className":42223,"style":19291},[420],[63,42225,42226,42229],{"style":19294},[63,42227],{"className":42228,"style":493},[428],[63,42230,42232],{"className":42231},[433,497,498,436],[63,42233,42235],{"className":42234},[399,436],[63,42236,19131],{"className":42237},[399,436],[63,42239,507],{"className":42240},[506],[63,42242,42244],{"className":42243},[416],[63,42245,42247],{"className":42246,"style":1554},[420],[63,42248],{},[63,42250,1078],{"className":42251},[403],[63,42253,42255,42258],{"className":42254},[399],[63,42256,38],{"className":42257},[399,472],[63,42259,42261],{"className":42260},[476],[63,42262,42264,42295],{"className":42263},[411,412],[63,42265,42267,42292],{"className":42266},[416],[63,42268,42270,42281],{"className":42269,"style":37441},[420],[63,42271,42272,42275],{"style":17457},[63,42273],{"className":42274,"style":817},[428],[63,42276,42278],{"className":42277},[433,434,435,436],[63,42279,612],{"className":42280},[399,472,436],[63,42282,42283,42286],{"style":16675},[63,42284],{"className":42285,"style":817},[428],[63,42287,42289],{"className":42288},[433,434,435,436],[63,42290,1057],{"className":42291},[399,472,436],[63,42293,507],{"className":42294},[506],[63,42296,42298],{"className":42297},[416],[63,42299,42301],{"className":42300,"style":17487},[420],[63,42302],{},[63,42304,13030],{"className":42305},[399],[63,42307,42309,42312],{"className":42308},[399],[63,42310,12890],{"className":42311},[399,472],[63,42313,42315],{"className":42314},[476],[63,42316,42318,42349],{"className":42317},[411,412],[63,42319,42321,42346],{"className":42320},[416],[63,42322,42324,42335],{"className":42323,"style":37441},[420],[63,42325,42326,42329],{"style":17457},[63,42327],{"className":42328,"style":817},[428],[63,42330,42332],{"className":42331},[433,434,435,436],[63,42333,612],{"className":42334},[399,472,436],[63,42336,42337,42340],{"style":16675},[63,42338],{"className":42339,"style":817},[428],[63,42341,42343],{"className":42342},[433,434,435,436],[63,42344,1057],{"className":42345},[399,472,436],[63,42347,507],{"className":42348},[506],[63,42350,42352],{"className":42351},[416],[63,42353,42355],{"className":42354,"style":17487},[420],[63,42356],{},[63,42358,1092],{"className":42359},[593],[63,42361,42362,42365],{"style":554},[63,42363],{"className":42364,"style":429},[428],[63,42366],{"className":42367,"style":562},[561],[63,42369,42370,42373],{"style":16030},[63,42371],{"className":42372,"style":429},[428],[63,42374,42376,42419,42422,42473,42476,42527],{"className":42375},[399],[63,42377,42379,42382],{"className":42378},[399],[63,42380,5685],{"className":42381,"style":3251},[399,472],[63,42383,42385],{"className":42384},[476],[63,42386,42388,42411],{"className":42387},[411,412],[63,42389,42391,42408],{"className":42390},[416],[63,42392,42394],{"className":42393,"style":1532},[420],[63,42395,42396,42399],{"style":7483},[63,42397],{"className":42398,"style":817},[428],[63,42400,42402],{"className":42401},[433,434,435,436],[63,42403,42405],{"className":42404},[399,436],[63,42406,14101],{"className":42407,"style":3276},[399,472,436],[63,42409,507],{"className":42410},[506],[63,42412,42414],{"className":42413},[416],[63,42415,42417],{"className":42416,"style":1554},[420],[63,42418],{},[63,42420,1078],{"className":42421},[403],[63,42423,42425,42428],{"className":42424},[399],[63,42426,38],{"className":42427},[399,472],[63,42429,42431],{"className":42430},[476],[63,42432,42434,42465],{"className":42433},[411,412],[63,42435,42437,42462],{"className":42436},[416],[63,42438,42440,42451],{"className":42439,"style":18996},[420],[63,42441,42442,42445],{"style":17457},[63,42443],{"className":42444,"style":817},[428],[63,42446,42448],{"className":42447},[433,434,435,436],[63,42449,612],{"className":42450},[399,472,436],[63,42452,42453,42456],{"style":813},[63,42454],{"className":42455,"style":817},[428],[63,42457,42459],{"className":42458},[433,434,435,436],[63,42460,1057],{"className":42461},[399,472,436],[63,42463,507],{"className":42464},[506],[63,42466,42468],{"className":42467},[416],[63,42469,42471],{"className":42470,"style":17487},[420],[63,42472],{},[63,42474,13030],{"className":42475},[399],[63,42477,42479,42482],{"className":42478},[399],[63,42480,12890],{"className":42481},[399,472],[63,42483,42485],{"className":42484},[476],[63,42486,42488,42519],{"className":42487},[411,412],[63,42489,42491,42516],{"className":42490},[416],[63,42492,42494,42505],{"className":42493,"style":18996},[420],[63,42495,42496,42499],{"style":17457},[63,42497],{"className":42498,"style":817},[428],[63,42500,42502],{"className":42501},[433,434,435,436],[63,42503,612],{"className":42504},[399,472,436],[63,42506,42507,42510],{"style":813},[63,42508],{"className":42509,"style":817},[428],[63,42511,42513],{"className":42512},[433,434,435,436],[63,42514,1057],{"className":42515},[399,472,436],[63,42517,507],{"className":42518},[506],[63,42520,42522],{"className":42521},[416],[63,42523,42525],{"className":42524,"style":17487},[420],[63,42526],{},[63,42528,1092],{"className":42529},[593],[63,42531,507],{"className":42532},[506],[63,42534,42536],{"className":42535},[416],[63,42537,42539],{"className":42538,"style":16095},[420],[63,42540],{},[63,42542],{"className":42543},[593,404],[63,42545],{"className":42546,"style":1920},[776],[63,42548,5839],{"className":42549},[830],[63,42551],{"className":42552,"style":1920},[776],[63,42554,42556,42559,42562,42565,42572,42575,42618,42621,42624,42696],{"className":42555},[390],[63,42557],{"className":42558,"style":1155},[394],[63,42560,41360],{"className":42561,"style":41377},[399,472],[63,42563],{"className":42564,"style":1177},[776],[63,42566,42568],{"className":42567},[399],[63,42569,42571],{"className":42570},[399,1162],"KL",[63,42573,1078],{"className":42574},[403],[63,42576,42578,42581],{"className":42577},[399],[63,42579,5685],{"className":42580,"style":3251},[399,472],[63,42582,42584],{"className":42583},[476],[63,42585,42587,42610],{"className":42586},[411,412],[63,42588,42590,42607],{"className":42589},[416],[63,42591,42593],{"className":42592,"style":1532},[420],[63,42594,42595,42598],{"style":7483},[63,42596],{"className":42597,"style":817},[428],[63,42599,42601],{"className":42600},[433,434,435,436],[63,42602,42604],{"className":42603},[399,436],[63,42605,14101],{"className":42606,"style":3276},[399,472,436],[63,42608,507],{"className":42609},[506],[63,42611,42613],{"className":42612},[416],[63,42614,42616],{"className":42615,"style":1554},[420],[63,42617],{},[63,42619,1083],{"className":42620},[1173],[63,42622],{"className":42623,"style":1177},[776],[63,42625,42627,42630],{"className":42626},[399],[63,42628,5685],{"className":42629,"style":3251},[399,472],[63,42631,42633],{"className":42632},[476],[63,42634,42636,42688],{"className":42635},[411,412],[63,42637,42639,42685],{"className":42638},[416],[63,42640,42642],{"className":42641,"style":1532},[420],[63,42643,42644,42647],{"style":7483},[63,42645],{"className":42646,"style":817},[428],[63,42648,42650],{"className":42649},[433,434,435,436],[63,42651,42653],{"className":42652},[399,436],[63,42654,42656,42659],{"className":42655},[399,436],[63,42657,14101],{"className":42658,"style":3276},[399,472,436],[63,42660,42662],{"className":42661},[476],[63,42663,42665],{"className":42664},[411],[63,42666,42668],{"className":42667},[416],[63,42669,42671],{"className":42670,"style":19291},[420],[63,42672,42673,42676],{"style":19294},[63,42674],{"className":42675,"style":493},[428],[63,42677,42679],{"className":42678},[433,497,498,436],[63,42680,42682],{"className":42681},[399,436],[63,42683,19131],{"className":42684},[399,436],[63,42686,507],{"className":42687},[506],[63,42689,42691],{"className":42690},[416],[63,42692,42694],{"className":42693,"style":1554},[420],[63,42695],{},[63,42697,1092],{"className":42698},[593],[41339,42700,42702],{"id":42701},"ppo-clip截断重要性采样比","PPO-Clip：截断重要性采样比",[14,42704,42705,42706,42709,42710,12841],{},"PPO 还有一种更常用的实现，用 ",[11217,42707,42708],{},"截断（clip）重要性采样比"," 来代替 KL 惩罚，同样起到限制新旧策略偏差的作用。它的目标函数由两部分组成（下方红色与蓝色项），最终取两者的 ",[11217,42711,42712],{},"较小值",[12764,42714,42715,43065],{},[12767,42716,42717,42720,42721,42773,42774,43064],{},[11217,42718,42719],{},"红色部分","：原始的 surrogate 目标 ",[63,42722,42724,42743],{"className":42723},[342],[63,42725,42727],{"className":42726},[346],[348,42728,42729],{"xmlns":350},[352,42730,42731,42740],{},[355,42732,42733,42736,42738],{},[371,42734,42735],{},"ρ",[738,42737,1864],{},[371,42739,1054],{},[378,42741,42742],{"encoding":380},"\\rho \\cdot A",[63,42744,42746,42764],{"className":42745,"ariaHidden":386},[385],[63,42747,42749,42752,42755,42758,42761],{"className":42748},[390],[63,42750],{"className":42751,"style":1913},[394],[63,42753,42735],{"className":42754},[399,472],[63,42756],{"className":42757,"style":1920},[776],[63,42759,1864],{"className":42760},[830],[63,42762],{"className":42763,"style":1920},[776],[63,42765,42767,42770],{"className":42766},[390],[63,42768],{"className":42769,"style":18606},[394],[63,42771,1054],{"className":42772},[399,472],"，其中 ",[63,42775,42777,42835],{"className":42776},[342],[63,42778,42780],{"className":42779},[346],[348,42781,42782],{"xmlns":350},[352,42783,42784,42832],{},[355,42785,42786,42788,42790],{},[371,42787,42735],{},[738,42789,204],{},[358,42791,42792,42810],{},[355,42793,42794,42800,42802,42804,42806,42808],{},[368,42795,42796,42798],{},[371,42797,5685],{},[371,42799,14101],{},[738,42801,1078],{"stretchy":1077},[371,42803,38],{},[738,42805,13030],{},[371,42807,12890],{},[738,42809,1092],{"stretchy":1077},[355,42811,42812,42822,42824,42826,42828,42830],{},[368,42813,42814,42816],{},[371,42815,5685],{},[742,42817,42818,42820],{},[371,42819,14101],{},[738,42821,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,42823,1078],{"stretchy":1077},[371,42825,38],{},[738,42827,13030],{},[371,42829,12890],{},[738,42831,1092],{"stretchy":1077},[378,42833,42834],{"encoding":380},"\\rho = \\frac{P_\\theta(a \\mid s)}{P_{\\theta'}(a \\mid s)}",[63,42836,42838,42856],{"className":42837,"ariaHidden":386},[385],[63,42839,42841,42844,42847,42850,42853],{"className":42840},[390],[63,42842],{"className":42843,"style":1754},[394],[63,42845,42735],{"className":42846},[399,472],[63,42848],{"className":42849,"style":777},[776],[63,42851,204],{"className":42852},[781],[63,42854],{"className":42855,"style":777},[776],[63,42857,42859,42863],{"className":42858},[390],[63,42860],{"className":42861,"style":42862},[394],"height:1.5384em;vertical-align:-0.5284em;",[63,42864,42866,42869,43061],{"className":42865},[399],[63,42867],{"className":42868},[403,404],[63,42870,42872],{"className":42871},[358],[63,42873,42875,43053],{"className":42874},[411,412],[63,42876,42878,43050],{"className":42877},[416],[63,42879,42881,42976,42984],{"className":42880,"style":32772},[420],[63,42882,42883,42886],{"style":18653},[63,42884],{"className":42885,"style":429},[428],[63,42887,42889],{"className":42888},[433,434,435,436],[63,42890,42892,42961,42964,42967,42970,42973],{"className":42891},[399,436],[63,42893,42895,42898],{"className":42894},[399,436],[63,42896,5685],{"className":42897,"style":3251},[399,472,436],[63,42899,42901],{"className":42900},[476],[63,42902,42904,42953],{"className":42903},[411,412],[63,42905,42907,42950],{"className":42906},[416],[63,42908,42910],{"className":42909,"style":486},[420],[63,42911,42912,42915],{"style":41207},[63,42913],{"className":42914,"style":41211},[428],[63,42916,42918],{"className":42917},[433,497,498,436],[63,42919,42921],{"className":42920},[399,436],[63,42922,42924,42927],{"className":42923},[399,436],[63,42925,14101],{"className":42926,"style":3276},[399,472,436],[63,42928,42930],{"className":42929},[476],[63,42931,42933],{"className":42932},[411],[63,42934,42936],{"className":42935},[416],[63,42937,42939],{"className":42938,"style":41236},[420],[63,42940,42941,42944],{"style":41239},[63,42942],{"className":42943,"style":41243},[428],[63,42945,42947],{"className":42946},[399,436],[63,42948,19131],{"className":42949},[399,436],[63,42951,507],{"className":42952},[506],[63,42954,42956],{"className":42955},[416],[63,42957,42959],{"className":42958,"style":41259},[420],[63,42960],{},[63,42962,1078],{"className":42963},[403,436],[63,42965,38],{"className":42966},[399,472,436],[63,42968,13030],{"className":42969},[781,436],[63,42971,12890],{"className":42972},[399,472,436],[63,42974,1092],{"className":42975},[593,436],[63,42977,42978,42981],{"style":554},[63,42979],{"className":42980,"style":429},[428],[63,42982],{"className":42983,"style":562},[561],[63,42985,42986,42989],{"style":32806},[63,42987],{"className":42988,"style":429},[428],[63,42990,42992],{"className":42991},[433,434,435,436],[63,42993,42995,43035,43038,43041,43044,43047],{"className":42994},[399,436],[63,42996,42998,43001],{"className":42997},[399,436],[63,42999,5685],{"className":43000,"style":3251},[399,472,436],[63,43002,43004],{"className":43003},[476],[63,43005,43007,43027],{"className":43006},[411,412],[63,43008,43010,43024],{"className":43009},[416],[63,43011,43013],{"className":43012,"style":486},[420],[63,43014,43015,43018],{"style":14320},[63,43016],{"className":43017,"style":493},[428],[63,43019,43021],{"className":43020},[433,497,498,436],[63,43022,14101],{"className":43023,"style":3276},[399,472,436],[63,43025,507],{"className":43026},[506],[63,43028,43030],{"className":43029},[416],[63,43031,43033],{"className":43032,"style":514},[420],[63,43034],{},[63,43036,1078],{"className":43037},[403,436],[63,43039,38],{"className":43040},[399,472,436],[63,43042,13030],{"className":43043},[781,436],[63,43045,12890],{"className":43046},[399,472,436],[63,43048,1092],{"className":43049},[593,436],[63,43051,507],{"className":43052},[506],[63,43054,43056],{"className":43055},[416],[63,43057,43059],{"className":43058,"style":41331},[420],[63,43060],{},[63,43062],{"className":43063},[593,404]," 是重要性采样比；",[12767,43066,43067,43070,43071,43100,43101,43204,43205,43233,43234,43262],{},[11217,43068,43069],{},"蓝色部分","：把 ",[63,43072,43074,43088],{"className":43073},[342],[63,43075,43077],{"className":43076},[346],[348,43078,43079],{"xmlns":350},[352,43080,43081,43085],{},[355,43082,43083],{},[371,43084,42735],{},[378,43086,43087],{"encoding":380},"\\rho",[63,43089,43091],{"className":43090,"ariaHidden":386},[385],[63,43092,43094,43097],{"className":43093},[390],[63,43095],{"className":43096,"style":1754},[394],[63,43098,42735],{"className":43099},[399,472]," 截断到 ",[63,43102,43104,43137],{"className":43103},[342],[63,43105,43107],{"className":43106},[346],[348,43108,43109],{"xmlns":350},[352,43110,43111,43134],{},[355,43112,43113,43115,43117,43119,43122,43124,43126,43128,43130,43132],{},[738,43114,11448],{"stretchy":1077},[361,43116,363],{},[738,43118,18511],{},[371,43120,43121],{},"ϵ",[738,43123,1083],{"separator":386},[3343,43125,7365],{},[361,43127,363],{},[738,43129,5839],{},[371,43131,43121],{},[738,43133,11763],{"stretchy":1077},[378,43135,43136],{"encoding":380},"[1-\\epsilon,\\, 1+\\epsilon]",[63,43138,43140,43161,43192],{"className":43139,"ariaHidden":386},[385],[63,43141,43143,43146,43149,43152,43155,43158],{"className":43142},[390],[63,43144],{"className":43145,"style":1155},[394],[63,43147,11448],{"className":43148},[403],[63,43150,363],{"className":43151},[399],[63,43153],{"className":43154,"style":1920},[776],[63,43156,18511],{"className":43157},[830],[63,43159],{"className":43160,"style":1920},[776],[63,43162,43164,43168,43171,43174,43177,43180,43183,43186,43189],{"className":43163},[390],[63,43165],{"className":43166,"style":43167},[394],"height:0.8389em;vertical-align:-0.1944em;",[63,43169,43121],{"className":43170},[399,472],[63,43172,1083],{"className":43173},[1173],[63,43175],{"className":43176,"style":1177},[776],[63,43178],{"className":43179,"style":1177},[776],[63,43181,363],{"className":43182},[399],[63,43184],{"className":43185,"style":1920},[776],[63,43187,5839],{"className":43188},[830],[63,43190],{"className":43191,"style":1920},[776],[63,43193,43195,43198,43201],{"className":43194},[390],[63,43196],{"className":43197,"style":1155},[394],[63,43199,43121],{"className":43200},[399,472],[63,43202,11763],{"className":43203},[593]," 区间内之后再乘以 ",[63,43206,43208,43221],{"className":43207},[342],[63,43209,43211],{"className":43210},[346],[348,43212,43213],{"xmlns":350},[352,43214,43215,43219],{},[355,43216,43217],{},[371,43218,1054],{},[378,43220,1054],{"encoding":380},[63,43222,43224],{"className":43223,"ariaHidden":386},[385],[63,43225,43227,43230],{"className":43226},[390],[63,43228],{"className":43229,"style":18606},[394],[63,43231,1054],{"className":43232},[399,472],"；当 ",[63,43235,43237,43250],{"className":43236},[342],[63,43238,43240],{"className":43239},[346],[348,43241,43242],{"xmlns":350},[352,43243,43244,43248],{},[355,43245,43246],{},[371,43247,42735],{},[378,43249,43087],{"encoding":380},[63,43251,43253],{"className":43252,"ariaHidden":386},[385],[63,43254,43256,43259],{"className":43255},[390],[63,43257],{"className":43258,"style":1754},[394],[63,43260,42735],{"className":43261},[399,472]," 落在区间内时返回原值，落在外面则返回最近的边界值。",[14,43264,43265,43266,43317,43318,43346,43347,43317,43398,43426],{},"对二者取 min 的目的是：当一个动作有正向优势 ",[63,43267,43269,43287],{"className":43268},[342],[63,43270,43272],{"className":43271},[346],[348,43273,43274],{"xmlns":350},[352,43275,43276,43284],{},[355,43277,43278,43280,43282],{},[371,43279,1054],{},[738,43281,11169],{},[361,43283,1802],{},[378,43285,43286],{"encoding":380},"A > 0",[63,43288,43290,43308],{"className":43289,"ariaHidden":386},[385],[63,43291,43293,43296,43299,43302,43305],{"className":43292},[390],[63,43294],{"className":43295,"style":876},[394],[63,43297,1054],{"className":43298},[399,472],[63,43300],{"className":43301,"style":777},[776],[63,43303,11169],{"className":43304},[781],[63,43306],{"className":43307,"style":777},[776],[63,43309,43311,43314],{"className":43310},[390],[63,43312],{"className":43313,"style":1814},[394],[63,43315,1802],{"className":43316},[399]," 时，限制 ",[63,43319,43321,43334],{"className":43320},[342],[63,43322,43324],{"className":43323},[346],[348,43325,43326],{"xmlns":350},[352,43327,43328,43332],{},[355,43329,43330],{},[371,43331,42735],{},[378,43333,43087],{"encoding":380},[63,43335,43337],{"className":43336,"ariaHidden":386},[385],[63,43338,43340,43343],{"className":43339},[390],[63,43341],{"className":43342,"style":1754},[394],[63,43344,42735],{"className":43345},[399,472]," 不会被推得过高（避免步子迈太大）；当 ",[63,43348,43350,43368],{"className":43349},[342],[63,43351,43353],{"className":43352},[346],[348,43354,43355],{"xmlns":350},[352,43356,43357,43365],{},[355,43358,43359,43361,43363],{},[371,43360,1054],{},[738,43362,10840],{},[361,43364,1802],{},[378,43366,43367],{"encoding":380},"A \u003C 0",[63,43369,43371,43389],{"className":43370,"ariaHidden":386},[385],[63,43372,43374,43377,43380,43383,43386],{"className":43373},[390],[63,43375],{"className":43376,"style":876},[394],[63,43378,1054],{"className":43379},[399,472],[63,43381],{"className":43382,"style":777},[776],[63,43384,10840],{"className":43385},[781],[63,43387],{"className":43388,"style":777},[776],[63,43390,43392,43395],{"className":43391},[390],[63,43393],{"className":43394,"style":1814},[394],[63,43396,1802],{"className":43397},[399],[63,43399,43401,43414],{"className":43400},[342],[63,43402,43404],{"className":43403},[346],[348,43405,43406],{"xmlns":350},[352,43407,43408,43412],{},[355,43409,43410],{},[371,43411,42735],{},[378,43413,43087],{"encoding":380},[63,43415,43417],{"className":43416,"ariaHidden":386},[385],[63,43418,43420,43423],{"className":43419},[390],[63,43421],{"className":43422,"style":1754},[394],[63,43424,42735],{"className":43425},[399,472]," 不会被压得过低。这样既保证了\"敢于改进策略\"，又避免了\"一次更新偏离太多\"。",[63,43428,43430],{"className":43429},[14495],[63,43431,43433,43769],{"className":43432},[342],[63,43434,43436],{"className":43435},[346],[348,43437,43438],{"xmlns":350,"display":9561},[352,43439,43440,43766],{},[355,43441,43442,43444,43446,43448,43455,43457,43459,43465,43479,43497,43500,43502],{},[371,43443,7842],{},[371,43445,1072],{},[371,43447,12890],{},[368,43449,43450,43452],{},[371,43451,12890],{},[3343,43453,43454],{},"ppo2",[738,43456,204],{},[738,43458,18511],{},[358,43460,43461,43463],{},[361,43462,363],{},[371,43464,7342],{},[14828,43466,43467,43469,43477],{},[738,43468,1874],{},[355,43470,43471,43473,43475],{},[371,43472,612],{},[738,43474,204],{},[361,43476,363],{},[371,43478,7342],{},[14828,43480,43481,43483,43491],{},[738,43482,1874],{},[355,43484,43485,43487,43489],{},[371,43486,1057],{},[738,43488,204],{},[361,43490,363],{},[368,43492,43493,43495],{},[371,43494,13358],{},[371,43496,612],{},[371,43498,43499],{},"min",[738,43501,1100],{},[355,43503,43504,43506,43615,43617,43619,43764],{},[738,43505,1078],{"fence":386},[14519,43507,43509,43527,43529,43537,43539,43547,43549],{"mathcolor":43508},"red",[1870,43510,43511,43513,43519],{},[371,43512,1054],{},[742,43514,43515,43517],{},[371,43516,14101],{},[738,43518,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,43520,43521,43523,43525],{},[371,43522,27970],{},[371,43524,1054],{},[371,43526,5688],{},[738,43528,1078],{"stretchy":1077},[1870,43530,43531,43533,43535],{},[371,43532,12890],{},[371,43534,612],{},[371,43536,1057],{},[738,43538,1083],{"separator":386},[1870,43540,43541,43543,43545],{},[371,43542,38],{},[371,43544,612],{},[371,43546,1057],{},[738,43548,1092],{"stretchy":1077},[358,43550,43551,43581],{},[355,43552,43553,43559,43561,43569,43571,43579],{},[368,43554,43555,43557],{},[371,43556,5685],{},[371,43558,14101],{},[738,43560,1078],{"stretchy":1077},[1870,43562,43563,43565,43567],{},[371,43564,38],{},[371,43566,612],{},[371,43568,1057],{},[371,43570,13030],{"mathvariant":1053},[1870,43572,43573,43575,43577],{},[371,43574,12890],{},[371,43576,612],{},[371,43578,1057],{},[738,43580,1092],{"stretchy":1077},[355,43582,43583,43593,43595,43603,43605,43613],{},[368,43584,43585,43587],{},[371,43586,5685],{},[742,43588,43589,43591],{},[371,43590,14101],{},[738,43592,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,43594,1078],{"stretchy":1077},[1870,43596,43597,43599,43601],{},[371,43598,38],{},[371,43600,612],{},[371,43602,1057],{},[371,43604,13030],{"mathvariant":1053},[1870,43606,43607,43609,43611],{},[371,43608,12890],{},[371,43610,612],{},[371,43612,1057],{},[738,43614,1092],{"stretchy":1077},[738,43616,1083],{"separator":386},[3343,43618,7365],{},[14519,43620,43622,43632,43724,43742,43744,43752,43754,43762],{"mathcolor":43621},"skyblue",[355,43623,43624,43626,43628,43630],{},[371,43625,3010],{"mathvariant":1053},[371,43627,2969],{"mathvariant":1053},[371,43629,1069],{"mathvariant":1053},[371,43631,14],{"mathvariant":1053},[355,43633,43634,43636,43702,43704,43706,43708,43710,43712,43714,43716,43718,43720,43722],{},[738,43635,1078],{"fence":386},[358,43637,43638,43668],{},[355,43639,43640,43646,43648,43656,43658,43666],{},[368,43641,43642,43644],{},[371,43643,5685],{},[371,43645,14101],{},[738,43647,1078],{"stretchy":1077},[1870,43649,43650,43652,43654],{},[371,43651,38],{},[371,43653,612],{},[371,43655,1057],{},[371,43657,13030],{"mathvariant":1053},[1870,43659,43660,43662,43664],{},[371,43661,12890],{},[371,43663,612],{},[371,43665,1057],{},[738,43667,1092],{"stretchy":1077},[355,43669,43670,43680,43682,43690,43692,43700],{},[368,43671,43672,43674],{},[371,43673,5685],{},[742,43675,43676,43678],{},[371,43677,14101],{},[738,43679,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[738,43681,1078],{"stretchy":1077},[1870,43683,43684,43686,43688],{},[371,43685,38],{},[371,43687,612],{},[371,43689,1057],{},[371,43691,13030],{"mathvariant":1053},[1870,43693,43694,43696,43698],{},[371,43695,12890],{},[371,43697,612],{},[371,43699,1057],{},[738,43701,1092],{"stretchy":1077},[738,43703,1083],{"separator":386},[3343,43705,7365],{},[361,43707,363],{},[738,43709,18511],{},[371,43711,43121],{},[738,43713,1083],{"separator":386},[3343,43715,7365],{},[361,43717,363],{},[738,43719,5839],{},[371,43721,43121],{},[738,43723,1092],{"fence":386},[1870,43725,43726,43728,43734],{},[371,43727,1054],{},[742,43729,43730,43732],{},[371,43731,14101],{},[738,43733,19131],{"mathvariant":1053,"lspace":14511,"rspace":14511},[355,43735,43736,43738,43740],{},[371,43737,27970],{},[371,43739,1054],{},[371,43741,5688],{},[738,43743,1078],{"stretchy":1077},[1870,43745,43746,43748,43750],{},[371,43747,12890],{},[371,43749,612],{},[371,43751,1057],{},[738,43753,1083],{"separator":386},[1870,43755,43756,43758,43760],{},[371,43757,38],{},[371,43759,612],{},[371,43761,1057],{},[738,43763,1092],{"stretchy":1077},[738,43765,1092],{"fence":386},[378,43767,43768],{"encoding":380},"Loss_{\\text{ppo2}} = -\\frac{1}{N} \\sum_{n=1}^{N} \\sum_{t=1}^{T_n} \\min\\left(\\textcolor{red}{A_{\\theta'}^{GAE}(s_n^t, a_n^t) \\frac{P_{\\theta}(a_n^t|s_n^t)}{P_{\\theta'}(a_n^t|s_n^t)}},\\,\\textcolor{skyblue}{\\mathrm{clip}\\left(\\frac{P_{\\theta}(a_n^t|s_n^t)}{P_{\\theta'}(a_n^t|s_n^t)},\\, 1-\\epsilon,\\, 1+\\epsilon\\right) A_{\\theta'}^{GAE}(s_n^t, a_n^t)}\\right)",[63,43770,43772,43839],{"className":43771,"ariaHidden":386},[385],[63,43773,43775,43778,43781,43784,43830,43833,43836],{"className":43774},[390],[63,43776],{"className":43777,"style":41630},[394],[63,43779,7842],{"className":43780},[399,472],[63,43782,41637],{"className":43783},[399,472],[63,43785,43787,43790],{"className":43786},[399],[63,43788,12890],{"className":43789},[399,472],[63,43791,43793],{"className":43792},[476],[63,43794,43796,43822],{"className":43795},[411,412],[63,43797,43799,43819],{"className":43798},[416],[63,43800,43802],{"className":43801,"style":3157},[420],[63,43803,43804,43807],{"style":1535},[63,43805],{"className":43806,"style":817},[428],[63,43808,43810],{"className":43809},[433,434,435,436],[63,43811,43813],{"className":43812},[399,436],[63,43814,43816],{"className":43815},[399,5395,436],[63,43817,43454],{"className":43818},[399,436],[63,43820,507],{"className":43821},[506],[63,43823,43825],{"className":43824},[416],[63,43826,43828],{"className":43827,"style":41682},[420],[63,43829],{},[63,43831],{"className":43832,"style":777},[776],[63,43834,204],{"className":43835},[781],[63,43837],{"className":43838,"style":777},[776],[63,43840,43842,43845,43848,43910,43913,43980,43983,44087,44090,44093,44096],{"className":43841},[390],[63,43843],{"className":43844,"style":31737},[394],[63,43846,18511],{"className":43847},[399],[63,43849,43851,43854,43907],{"className":43850},[399],[63,43852],{"className":43853},[403,404],[63,43855,43857],{"className":43856},[358],[63,43858,43860,43899],{"className":43859},[411,412],[63,43861,43863,43896],{"className":43862},[416],[63,43864,43866,43877,43885],{"className":43865,"style":16424},[420],[63,43867,43868,43871],{"style":15964},[63,43869],{"className":43870,"style":429},[428],[63,43872,43874],{"className":43873},[399],[63,43875,7342],{"className":43876,"style":16436},[399,472],[63,43878,43879,43882],{"style":554},[63,43880],{"className":43881,"style":429},[428],[63,43883],{"className":43884,"style":562},[561],[63,43886,43887,43890],{"style":16030},[63,43888],{"className":43889,"style":429},[428],[63,43891,43893],{"className":43892},[399],[63,43894,363],{"className":43895},[399],[63,43897,507],{"className":43898},[506],[63,43900,43902],{"className":43901},[416],[63,43903,43905],{"className":43904,"style":16465},[420],[63,43906],{},[63,43908],{"className":43909},[593,404],[63,43911],{"className":43912,"style":1177},[776],[63,43914,43916],{"className":43915},[1212,14383],[63,43917,43919,43972],{"className":43918},[411,412],[63,43920,43922,43969],{"className":43921},[416],[63,43923,43925,43945,43955],{"className":43924,"style":16486},[420],[63,43926,43927,43930],{"style":16489},[63,43928],{"className":43929,"style":15577},[428],[63,43931,43933],{"className":43932},[433,434,435,436],[63,43934,43936,43939,43942],{"className":43935},[399,436],[63,43937,612],{"className":43938},[399,472,436],[63,43940,204],{"className":43941},[781,436],[63,43943,363],{"className":43944},[399,436],[63,43946,43947,43950],{"style":15586},[63,43948],{"className":43949,"style":15577},[428],[63,43951,43952],{},[63,43953,1874],{"className":43954},[1212,1958,15595],[63,43956,43957,43960],{"style":16520},[63,43958],{"className":43959,"style":15577},[428],[63,43961,43963],{"className":43962},[433,434,435,436],[63,43964,43966],{"className":43965},[399,436],[63,43967,7342],{"className":43968,"style":16436},[399,472,436],[63,43970,507],{"className":43971},[506],[63,43973,43975],{"className":43974},[416],[63,43976,43978],{"className":43977,"style":16539},[420],[63,43979],{},[63,43981],{"className":43982,"style":1177},[776],[63,43984,43986],{"className":43985},[1212,14383],[63,43987,43989,44079],{"className":43988},[411,412],[63,43990,43992,44076],{"className":43991},[416],[63,43993,43995,44015,44025],{"className":43994,"style":17295},[420],[63,43996,43997,44000],{"style":16489},[63,43998],{"className":43999,"style":15577},[428],[63,44001,44003],{"className":44002},[433,434,435,436],[63,44004,44006,44009,44012],{"className":44005},[399,436],[63,44007,1057],{"className":44008},[399,472,436],[63,44010,204],{"className":44011},[781,436],[63,44013,363],{"className":44014},[399,436],[63,44016,44017,44020],{"style":15586},[63,44018],{"className":44019,"style":15577},[428],[63,44021,44022],{},[63,44023,1874],{"className":44024},[1212,1958,15595],[63,44026,44027,44030],{"style":17328},[63,44028],{"className":44029,"style":15577},[428],[63,44031,44033],{"className":44032},[433,434,435,436],[63,44034,44036],{"className":44035},[399,436],[63,44037,44039,44042],{"className":44038},[399,436],[63,44040,13358],{"className":44041,"style":3251},[399,472,436],[63,44043,44045],{"className":44044},[476],[63,44046,44048,44068],{"className":44047},[411,412],[63,44049,44051,44065],{"className":44050},[416],[63,44052,44054],{"className":44053,"style":4367},[420],[63,44055,44056,44059],{"style":17358},[63,44057],{"className":44058,"style":493},[428],[63,44060,44062],{"className":44061},[433,497,498,436],[63,44063,612],{"className":44064},[399,472,436],[63,44066,507],{"className":44067},[506],[63,44069,44071],{"className":44070},[416],[63,44072,44074],{"className":44073,"style":4389},[420],[63,44075],{},[63,44077,507],{"className":44078},[506],[63,44080,44082],{"className":44081},[416],[63,44083,44085],{"className":44084,"style":16539},[420],[63,44086],{},[63,44088],{"className":44089,"style":1177},[776],[63,44091,43499],{"className":44092},[1212],[63,44094],{"className":44095,"style":1177},[776],[63,44097,44099,44105,44200,44203,44254,44257,44260,44311,44314,44709,44712,44715,44718,44726,44729,45195,45198,45291,45294,45345,45348,45351,45402,45405],{"className":44098},[1222],[63,44100,44102],{"className":44101,"style":1227},[403,1226],[63,44103,1078],{"className":44104},[1231,435],[63,44106,44109,44112],{"className":44107,"style":44108},[399],"color:red;",[63,44110,1054],{"className":44111,"style":44108},[399,472],[63,44113,44115],{"className":44114},[476],[63,44116,44118,44192],{"className":44117},[411,412],[63,44119,44121,44189],{"className":44120},[416],[63,44122,44124,44168],{"className":44123,"style":25181},[420],[63,44125,44126,44129],{"style":17457},[63,44127],{"className":44128,"style":817},[428],[63,44130,44132],{"className":44131,"style":44108},[433,434,435,436],[63,44133,44135],{"className":44134,"style":44108},[399,436],[63,44136,44138,44142],{"className":44137,"style":44108},[399,436],[63,44139,14101],{"className":44140,"style":44141},[399,472,436],"margin-right:0.02778em;color:red;",[63,44143,44145],{"className":44144},[476],[63,44146,44148],{"className":44147},[411],[63,44149,44151],{"className":44150},[416],[63,44152,44154],{"className":44153,"style":19291},[420],[63,44155,44156,44159],{"style":19294},[63,44157],{"className":44158,"style":493},[428],[63,44160,44162],{"className":44161,"style":44108},[433,497,498,436],[63,44163,44165],{"className":44164,"style":44108},[399,436],[63,44166,19131],{"className":44167,"style":44108},[399,436],[63,44169,44170,44173],{"style":16572},[63,44171],{"className":44172,"style":817},[428],[63,44174,44176],{"className":44175,"style":44108},[433,434,435,436],[63,44177,44179,44182,44185],{"className":44178,"style":44108},[399,436],[63,44180,27970],{"className":44181,"style":44108},[399,472,436],[63,44183,1054],{"className":44184,"style":44108},[399,472,436],[63,44186,5688],{"className":44187,"style":44188},[399,472,436],"margin-right:0.05764em;color:red;",[63,44190,507],{"className":44191},[506],[63,44193,44195],{"className":44194},[416],[63,44196,44198],{"className":44197,"style":17487},[420],[63,44199],{},[63,44201,1078],{"className":44202,"style":44108},[403],[63,44204,44206,44209],{"className":44205,"style":44108},[399],[63,44207,12890],{"className":44208,"style":44108},[399,472],[63,44210,44212],{"className":44211},[476],[63,44213,44215,44246],{"className":44214},[411,412],[63,44216,44218,44243],{"className":44217},[416],[63,44219,44221,44232],{"className":44220,"style":17454},[420],[63,44222,44223,44226],{"style":17457},[63,44224],{"className":44225,"style":817},[428],[63,44227,44229],{"className":44228,"style":44108},[433,434,435,436],[63,44230,612],{"className":44231,"style":44108},[399,472,436],[63,44233,44234,44237],{"style":16572},[63,44235],{"className":44236,"style":817},[428],[63,44238,44240],{"className":44239,"style":44108},[433,434,435,436],[63,44241,1057],{"className":44242,"style":44108},[399,472,436],[63,44244,507],{"className":44245},[506],[63,44247,44249],{"className":44248},[416],[63,44250,44252],{"className":44251,"style":17487},[420],[63,44253],{},[63,44255,1083],{"className":44256,"style":44108},[1173],[63,44258],{"className":44259,"style":1177},[776],[63,44261,44263,44266],{"className":44262,"style":44108},[399],[63,44264,38],{"className":44265,"style":44108},[399,472],[63,44267,44269],{"className":44268},[476],[63,44270,44272,44303],{"className":44271},[411,412],[63,44273,44275,44300],{"className":44274},[416],[63,44276,44278,44289],{"className":44277,"style":17454},[420],[63,44279,44280,44283],{"style":17457},[63,44281],{"className":44282,"style":817},[428],[63,44284,44286],{"className":44285,"style":44108},[433,434,435,436],[63,44287,612],{"className":44288,"style":44108},[399,472,436],[63,44290,44291,44294],{"style":16572},[63,44292],{"className":44293,"style":817},[428],[63,44295,44297],{"className":44296,"style":44108},[433,434,435,436],[63,44298,1057],{"className":44299,"style":44108},[399,472,436],[63,44301,507],{"className":44302},[506],[63,44304,44306],{"className":44305},[416],[63,44307,44309],{"className":44308,"style":17487},[420],[63,44310],{},[63,44312,1092],{"className":44313,"style":44108},[593],[63,44315,44317,44320,44706],{"className":44316,"style":44108},[399],[63,44318],{"className":44319},[403,404],[63,44321,44323],{"className":44322},[358],[63,44324,44326,44698],{"className":44325},[411,412],[63,44327,44329,44695],{"className":44328},[416],[63,44330,44332,44524,44533],{"className":44331,"style":37339},[420],[63,44333,44334,44337],{"style":15964},[63,44335],{"className":44336,"style":429},[428],[63,44338,44340,44413,44416,44467,44470,44521],{"className":44339,"style":44108},[399],[63,44341,44343,44347],{"className":44342,"style":44108},[399],[63,44344,5685],{"className":44345,"style":44346},[399,472],"margin-right:0.13889em;color:red;",[63,44348,44350],{"className":44349},[476],[63,44351,44353,44405],{"className":44352},[411,412],[63,44354,44356,44402],{"className":44355},[416],[63,44357,44359],{"className":44358,"style":1532},[420],[63,44360,44361,44364],{"style":7483},[63,44362],{"className":44363,"style":817},[428],[63,44365,44367],{"className":44366,"style":44108},[433,434,435,436],[63,44368,44370],{"className":44369,"style":44108},[399,436],[63,44371,44373,44376],{"className":44372,"style":44108},[399,436],[63,44374,14101],{"className":44375,"style":44141},[399,472,436],[63,44377,44379],{"className":44378},[476],[63,44380,44382],{"className":44381},[411],[63,44383,44385],{"className":44384},[416],[63,44386,44388],{"className":44387,"style":19291},[420],[63,44389,44390,44393],{"style":19294},[63,44391],{"className":44392,"style":493},[428],[63,44394,44396],{"className":44395,"style":44108},[433,497,498,436],[63,44397,44399],{"className":44398,"style":44108},[399,436],[63,44400,19131],{"className":44401,"style":44108},[399,436],[63,44403,507],{"className":44404},[506],[63,44406,44408],{"className":44407},[416],[63,44409,44411],{"className":44410,"style":1554},[420],[63,44412],{},[63,44414,1078],{"className":44415,"style":44108},[403],[63,44417,44419,44422],{"className":44418,"style":44108},[399],[63,44420,38],{"className":44421,"style":44108},[399,472],[63,44423,44425],{"className":44424},[476],[63,44426,44428,44459],{"className":44427},[411,412],[63,44429,44431,44456],{"className":44430},[416],[63,44432,44434,44445],{"className":44433,"style":37441},[420],[63,44435,44436,44439],{"style":17457},[63,44437],{"className":44438,"style":817},[428],[63,44440,44442],{"className":44441,"style":44108},[433,434,435,436],[63,44443,612],{"className":44444,"style":44108},[399,472,436],[63,44446,44447,44450],{"style":16675},[63,44448],{"className":44449,"style":817},[428],[63,44451,44453],{"className":44452,"style":44108},[433,434,435,436],[63,44454,1057],{"className":44455,"style":44108},[399,472,436],[63,44457,507],{"className":44458},[506],[63,44460,44462],{"className":44461},[416],[63,44463,44465],{"className":44464,"style":17487},[420],[63,44466],{},[63,44468,13030],{"className":44469,"style":44108},[399],[63,44471,44473,44476],{"className":44472,"style":44108},[399],[63,44474,12890],{"className":44475,"style":44108},[399,472],[63,44477,44479],{"className":44478},[476],[63,44480,44482,44513],{"className":44481},[411,412],[63,44483,44485,44510],{"className":44484},[416],[63,44486,44488,44499],{"className":44487,"style":37441},[420],[63,44489,44490,44493],{"style":17457},[63,44491],{"className":44492,"style":817},[428],[63,44494,44496],{"className":44495,"style":44108},[433,434,435,436],[63,44497,612],{"className":44498,"style":44108},[399,472,436],[63,44500,44501,44504],{"style":16675},[63,44502],{"className":44503,"style":817},[428],[63,44505,44507],{"className":44506,"style":44108},[433,434,435,436],[63,44508,1057],{"className":44509,"style":44108},[399,472,436],[63,44511,507],{"className":44512},[506],[63,44514,44516],{"className":44515},[416],[63,44517,44519],{"className":44518,"style":17487},[420],[63,44520],{},[63,44522,1092],{"className":44523,"style":44108},[593],[63,44525,44526,44529],{"style":554},[63,44527],{"className":44528,"style":429},[428],[63,44530],{"className":44531,"style":44532},[561],"color:red;border-bottom-width:0.04em;",[63,44534,44535,44538],{"style":16030},[63,44536],{"className":44537,"style":429},[428],[63,44539,44541,44584,44587,44638,44641,44692],{"className":44540,"style":44108},[399],[63,44542,44544,44547],{"className":44543,"style":44108},[399],[63,44545,5685],{"className":44546,"style":44346},[399,472],[63,44548,44550],{"className":44549},[476],[63,44551,44553,44576],{"className":44552},[411,412],[63,44554,44556,44573],{"className":44555},[416],[63,44557,44559],{"className":44558,"style":1532},[420],[63,44560,44561,44564],{"style":7483},[63,44562],{"className":44563,"style":817},[428],[63,44565,44567],{"className":44566,"style":44108},[433,434,435,436],[63,44568,44570],{"className":44569,"style":44108},[399,436],[63,44571,14101],{"className":44572,"style":44141},[399,472,436],[63,44574,507],{"className":44575},[506],[63,44577,44579],{"className":44578},[416],[63,44580,44582],{"className":44581,"style":1554},[420],[63,44583],{},[63,44585,1078],{"className":44586,"style":44108},[403],[63,44588,44590,44593],{"className":44589,"style":44108},[399],[63,44591,38],{"className":44592,"style":44108},[399,472],[63,44594,44596],{"className":44595},[476],[63,44597,44599,44630],{"className":44598},[411,412],[63,44600,44602,44627],{"className":44601},[416],[63,44603,44605,44616],{"className":44604,"style":18996},[420],[63,44606,44607,44610],{"style":17457},[63,44608],{"className":44609,"style":817},[428],[63,44611,44613],{"className":44612,"style":44108},[433,434,435,436],[63,44614,612],{"className":44615,"style":44108},[399,472,436],[63,44617,44618,44621],{"style":813},[63,44619],{"className":44620,"style":817},[428],[63,44622,44624],{"className":44623,"style":44108},[433,434,435,436],[63,44625,1057],{"className":44626,"style":44108},[399,472,436],[63,44628,507],{"className":44629},[506],[63,44631,44633],{"className":44632},[416],[63,44634,44636],{"className":44635,"style":17487},[420],[63,44637],{},[63,44639,13030],{"className":44640,"style":44108},[399],[63,44642,44644,44647],{"className":44643,"style":44108},[399],[63,44645,12890],{"className":44646,"style":44108},[399,472],[63,44648,44650],{"className":44649},[476],[63,44651,44653,44684],{"className":44652},[411,412],[63,44654,44656,44681],{"className":44655},[416],[63,44657,44659,44670],{"className":44658,"style":18996},[420],[63,44660,44661,44664],{"style":17457},[63,44662],{"className":44663,"style":817},[428],[63,44665,44667],{"className":44666,"style":44108},[433,434,435,436],[63,44668,612],{"className":44669,"style":44108},[399,472,436],[63,44671,44672,44675],{"style":813},[63,44673],{"className":44674,"style":817},[428],[63,44676,44678],{"className":44677,"style":44108},[433,434,435,436],[63,44679,1057],{"className":44680,"style":44108},[399,472,436],[63,44682,507],{"className":44683},[506],[63,44685,44687],{"className":44686},[416],[63,44688,44690],{"className":44689,"style":17487},[420],[63,44691],{},[63,44693,1092],{"className":44694,"style":44108},[593],[63,44696,507],{"className":44697},[506],[63,44699,44701],{"className":44700},[416],[63,44702,44704],{"className":44703,"style":16095},[420],[63,44705],{},[63,44707],{"className":44708},[593,404],[63,44710,1083],{"className":44711},[1173],[63,44713],{"className":44714,"style":1177},[776],[63,44716],{"className":44717,"style":1177},[776],[63,44719,44722],{"className":44720,"style":44721},[399],"color:skyblue;",[63,44723,44725],{"className":44724,"style":44721},[399,1162],"clip",[63,44727],{"className":44728,"style":1177},[776],[63,44730,44732,44741,45137,45140,45144,45147,45150,45154,45157,45160,45163,45166,45169,45172,45175,45178,45181,45184,45187],{"className":44731,"style":44721},[1222],[63,44733,44736],{"className":44734,"style":44735},[403,1226],"color:skyblue;top:0em;",[63,44737,44739],{"className":44738,"style":44721},[1231,435],[63,44740,1078],{"style":44721},[63,44742,44744,44747,45134],{"className":44743,"style":44721},[399],[63,44745],{"className":44746},[403,404],[63,44748,44750],{"className":44749},[358],[63,44751,44753,45126],{"className":44752},[411,412],[63,44754,44756,45123],{"className":44755},[416],[63,44757,44759,44952,44961],{"className":44758,"style":37339},[420],[63,44760,44761,44764],{"style":15964},[63,44762],{"className":44763,"style":429},[428],[63,44765,44767,44841,44844,44895,44898,44949],{"className":44766,"style":44721},[399],[63,44768,44770,44774],{"className":44769,"style":44721},[399],[63,44771,5685],{"className":44772,"style":44773},[399,472],"margin-right:0.13889em;color:skyblue;",[63,44775,44777],{"className":44776},[476],[63,44778,44780,44833],{"className":44779},[411,412],[63,44781,44783,44830],{"className":44782},[416],[63,44784,44786],{"className":44785,"style":1532},[420],[63,44787,44788,44791],{"style":7483},[63,44789],{"className":44790,"style":817},[428],[63,44792,44794],{"className":44793,"style":44721},[433,434,435,436],[63,44795,44797],{"className":44796,"style":44721},[399,436],[63,44798,44800,44804],{"className":44799,"style":44721},[399,436],[63,44801,14101],{"className":44802,"style":44803},[399,472,436],"margin-right:0.02778em;color:skyblue;",[63,44805,44807],{"className":44806},[476],[63,44808,44810],{"className":44809},[411],[63,44811,44813],{"className":44812},[416],[63,44814,44816],{"className":44815,"style":19291},[420],[63,44817,44818,44821],{"style":19294},[63,44819],{"className":44820,"style":493},[428],[63,44822,44824],{"className":44823,"style":44721},[433,497,498,436],[63,44825,44827],{"className":44826,"style":44721},[399,436],[63,44828,19131],{"className":44829,"style":44721},[399,436],[63,44831,507],{"className":44832},[506],[63,44834,44836],{"className":44835},[416],[63,44837,44839],{"className":44838,"style":1554},[420],[63,44840],{},[63,44842,1078],{"className":44843,"style":44721},[403],[63,44845,44847,44850],{"className":44846,"style":44721},[399],[63,44848,38],{"className":44849,"style":44721},[399,472],[63,44851,44853],{"className":44852},[476],[63,44854,44856,44887],{"className":44855},[411,412],[63,44857,44859,44884],{"className":44858},[416],[63,44860,44862,44873],{"className":44861,"style":37441},[420],[63,44863,44864,44867],{"style":17457},[63,44865],{"className":44866,"style":817},[428],[63,44868,44870],{"className":44869,"style":44721},[433,434,435,436],[63,44871,612],{"className":44872,"style":44721},[399,472,436],[63,44874,44875,44878],{"style":16675},[63,44876],{"className":44877,"style":817},[428],[63,44879,44881],{"className":44880,"style":44721},[433,434,435,436],[63,44882,1057],{"className":44883,"style":44721},[399,472,436],[63,44885,507],{"className":44886},[506],[63,44888,44890],{"className":44889},[416],[63,44891,44893],{"className":44892,"style":17487},[420],[63,44894],{},[63,44896,13030],{"className":44897,"style":44721},[399],[63,44899,44901,44904],{"className":44900,"style":44721},[399],[63,44902,12890],{"className":44903,"style":44721},[399,472],[63,44905,44907],{"className":44906},[476],[63,44908,44910,44941],{"className":44909},[411,412],[63,44911,44913,44938],{"className":44912},[416],[63,44914,44916,44927],{"className":44915,"style":37441},[420],[63,44917,44918,44921],{"style":17457},[63,44919],{"className":44920,"style":817},[428],[63,44922,44924],{"className":44923,"style":44721},[433,434,435,436],[63,44925,612],{"className":44926,"style":44721},[399,472,436],[63,44928,44929,44932],{"style":16675},[63,44930],{"className":44931,"style":817},[428],[63,44933,44935],{"className":44934,"style":44721},[433,434,435,436],[63,44936,1057],{"className":44937,"style":44721},[399,472,436],[63,44939,507],{"className":44940},[506],[63,44942,44944],{"className":44943},[416],[63,44945,44947],{"className":44946,"style":17487},[420],[63,44948],{},[63,44950,1092],{"className":44951,"style":44721},[593],[63,44953,44954,44957],{"style":554},[63,44955],{"className":44956,"style":429},[428],[63,44958],{"className":44959,"style":44960},[561],"color:skyblue;border-bottom-width:0.04em;",[63,44962,44963,44966],{"style":16030},[63,44964],{"className":44965,"style":429},[428],[63,44967,44969,45012,45015,45066,45069,45120],{"className":44968,"style":44721},[399],[63,44970,44972,44975],{"className":44971,"style":44721},[399],[63,44973,5685],{"className":44974,"style":44773},[399,472],[63,44976,44978],{"className":44977},[476],[63,44979,44981,45004],{"className":44980},[411,412],[63,44982,44984,45001],{"className":44983},[416],[63,44985,44987],{"className":44986,"style":1532},[420],[63,44988,44989,44992],{"style":7483},[63,44990],{"className":44991,"style":817},[428],[63,44993,44995],{"className":44994,"style":44721},[433,434,435,436],[63,44996,44998],{"className":44997,"style":44721},[399,436],[63,44999,14101],{"className":45000,"style":44803},[399,472,436],[63,45002,507],{"className":45003},[506],[63,45005,45007],{"className":45006},[416],[63,45008,45010],{"className":45009,"style":1554},[420],[63,45011],{},[63,45013,1078],{"className":45014,"style":44721},[403],[63,45016,45018,45021],{"className":45017,"style":44721},[399],[63,45019,38],{"className":45020,"style":44721},[399,472],[63,45022,45024],{"className":45023},[476],[63,45025,45027,45058],{"className":45026},[411,412],[63,45028,45030,45055],{"className":45029},[416],[63,45031,45033,45044],{"className":45032,"style":18996},[420],[63,45034,45035,45038],{"style":17457},[63,45036],{"className":45037,"style":817},[428],[63,45039,45041],{"className":45040,"style":44721},[433,434,435,436],[63,45042,612],{"className":45043,"style":44721},[399,472,436],[63,45045,45046,45049],{"style":813},[63,45047],{"className":45048,"style":817},[428],[63,45050,45052],{"className":45051,"style":44721},[433,434,435,436],[63,45053,1057],{"className":45054,"style":44721},[399,472,436],[63,45056,507],{"className":45057},[506],[63,45059,45061],{"className":45060},[416],[63,45062,45064],{"className":45063,"style":17487},[420],[63,45065],{},[63,45067,13030],{"className":45068,"style":44721},[399],[63,45070,45072,45075],{"className":45071,"style":44721},[399],[63,45073,12890],{"className":45074,"style":44721},[399,472],[63,45076,45078],{"className":45077},[476],[63,45079,45081,45112],{"className":45080},[411,412],[63,45082,45084,45109],{"className":45083},[416],[63,45085,45087,45098],{"className":45086,"style":18996},[420],[63,45088,45089,45092],{"style":17457},[63,45090],{"className":45091,"style":817},[428],[63,45093,45095],{"className":45094,"style":44721},[433,434,435,436],[63,45096,612],{"className":45097,"style":44721},[399,472,436],[63,45099,45100,45103],{"style":813},[63,45101],{"className":45102,"style":817},[428],[63,45104,45106],{"className":45105,"style":44721},[433,434,435,436],[63,45107,1057],{"className":45108,"style":44721},[399,472,436],[63,45110,507],{"className":45111},[506],[63,45113,45115],{"className":45114},[416],[63,45116,45118],{"className":45117,"style":17487},[420],[63,45119],{},[63,45121,1092],{"className":45122,"style":44721},[593],[63,45124,507],{"className":45125},[506],[63,45127,45129],{"className":45128},[416],[63,45130,45132],{"className":45131,"style":16095},[420],[63,45133],{},[63,45135],{"className":45136},[593,404],[63,45138,1083],{"className":45139,"style":44721},[1173],[63,45141],{"className":45142,"style":45143},[776],"color:skyblue;margin-right:0.1667em;",[63,45145],{"className":45146,"style":45143},[776],[63,45148,363],{"className":45149,"style":44721},[399],[63,45151],{"className":45152,"style":45153},[776],"color:skyblue;margin-right:0.2222em;",[63,45155,18511],{"className":45156,"style":44721},[830],[63,45158],{"className":45159,"style":45153},[776],[63,45161,43121],{"className":45162,"style":44721},[399,472],[63,45164,1083],{"className":45165,"style":44721},[1173],[63,45167],{"className":45168,"style":45143},[776],[63,45170],{"className":45171,"style":45143},[776],[63,45173,363],{"className":45174,"style":44721},[399],[63,45176],{"className":45177,"style":45153},[776],[63,45179,5839],{"className":45180,"style":44721},[830],[63,45182],{"className":45183,"style":45153},[776],[63,45185,43121],{"className":45186,"style":44721},[399,472],[63,45188,45190],{"className":45189,"style":44735},[593,1226],[63,45191,45193],{"className":45192,"style":44721},[1231,435],[63,45194,1092],{"style":44721},[63,45196],{"className":45197,"style":1177},[776],[63,45199,45201,45204],{"className":45200,"style":44721},[399],[63,45202,1054],{"className":45203,"style":44721},[399,472],[63,45205,45207],{"className":45206},[476],[63,45208,45210,45283],{"className":45209},[411,412],[63,45211,45213,45280],{"className":45212},[416],[63,45214,45216,45259],{"className":45215,"style":25181},[420],[63,45217,45218,45221],{"style":17457},[63,45219],{"className":45220,"style":817},[428],[63,45222,45224],{"className":45223,"style":44721},[433,434,435,436],[63,45225,45227],{"className":45226,"style":44721},[399,436],[63,45228,45230,45233],{"className":45229,"style":44721},[399,436],[63,45231,14101],{"className":45232,"style":44803},[399,472,436],[63,45234,45236],{"className":45235},[476],[63,45237,45239],{"className":45238},[411],[63,45240,45242],{"className":45241},[416],[63,45243,45245],{"className":45244,"style":19291},[420],[63,45246,45247,45250],{"style":19294},[63,45248],{"className":45249,"style":493},[428],[63,45251,45253],{"className":45252,"style":44721},[433,497,498,436],[63,45254,45256],{"className":45255,"style":44721},[399,436],[63,45257,19131],{"className":45258,"style":44721},[399,436],[63,45260,45261,45264],{"style":16572},[63,45262],{"className":45263,"style":817},[428],[63,45265,45267],{"className":45266,"style":44721},[433,434,435,436],[63,45268,45270,45273,45276],{"className":45269,"style":44721},[399,436],[63,45271,27970],{"className":45272,"style":44721},[399,472,436],[63,45274,1054],{"className":45275,"style":44721},[399,472,436],[63,45277,5688],{"className":45278,"style":45279},[399,472,436],"margin-right:0.05764em;color:skyblue;",[63,45281,507],{"className":45282},[506],[63,45284,45286],{"className":45285},[416],[63,45287,45289],{"className":45288,"style":17487},[420],[63,45290],{},[63,45292,1078],{"className":45293,"style":44721},[403],[63,45295,45297,45300],{"className":45296,"style":44721},[399],[63,45298,12890],{"className":45299,"style":44721},[399,472],[63,45301,45303],{"className":45302},[476],[63,45304,45306,45337],{"className":45305},[411,412],[63,45307,45309,45334],{"className":45308},[416],[63,45310,45312,45323],{"className":45311,"style":17454},[420],[63,45313,45314,45317],{"style":17457},[63,45315],{"className":45316,"style":817},[428],[63,45318,45320],{"className":45319,"style":44721},[433,434,435,436],[63,45321,612],{"className":45322,"style":44721},[399,472,436],[63,45324,45325,45328],{"style":16572},[63,45326],{"className":45327,"style":817},[428],[63,45329,45331],{"className":45330,"style":44721},[433,434,435,436],[63,45332,1057],{"className":45333,"style":44721},[399,472,436],[63,45335,507],{"className":45336},[506],[63,45338,45340],{"className":45339},[416],[63,45341,45343],{"className":45342,"style":17487},[420],[63,45344],{},[63,45346,1083],{"className":45347,"style":44721},[1173],[63,45349],{"className":45350,"style":1177},[776],[63,45352,45354,45357],{"className":45353,"style":44721},[399],[63,45355,38],{"className":45356,"style":44721},[399,472],[63,45358,45360],{"className":45359},[476],[63,45361,45363,45394],{"className":45362},[411,412],[63,45364,45366,45391],{"className":45365},[416],[63,45367,45369,45380],{"className":45368,"style":17454},[420],[63,45370,45371,45374],{"style":17457},[63,45372],{"className":45373,"style":817},[428],[63,45375,45377],{"className":45376,"style":44721},[433,434,435,436],[63,45378,612],{"className":45379,"style":44721},[399,472,436],[63,45381,45382,45385],{"style":16572},[63,45383],{"className":45384,"style":817},[428],[63,45386,45388],{"className":45387,"style":44721},[433,434,435,436],[63,45389,1057],{"className":45390,"style":44721},[399,472,436],[63,45392,507],{"className":45393},[506],[63,45395,45397],{"className":45396},[416],[63,45398,45400],{"className":45399,"style":17487},[420],[63,45401],{},[63,45403,1092],{"className":45404,"style":44721},[593],[63,45406,45408],{"className":45407,"style":1227},[593,1226],[63,45409,1092],{"className":45410},[1231,435],[14,45412,45413],{},"PPO-Clip 由于实现简单、效果稳定，是当前应用最广泛的版本，也是 RLHF/GRPO 等大模型对齐算法的基础。",[10,45415,45416],{"id":45416},"参考资料",[14,45418,45419],{},[38,45420,45423],{"href":45421,"rel":45422},"https://www.bilibili.com/video/BV1iz421h7gb/",[42],"零基础学习强化学习算法：ppo",{"title":59,"searchDepth":88,"depth":88,"links":45425},[45426,45427,45428,45429,45432,45436],{"id":12753,"depth":88,"text":12753},{"id":12759,"depth":88,"text":12759},{"id":14079,"depth":88,"text":14079},{"id":19608,"depth":88,"text":19609,"children":45430},[45431],{"id":27838,"depth":95,"text":27839},{"id":32394,"depth":88,"text":32395,"children":45433},[45434,45435],{"id":32592,"depth":95,"text":32592},{"id":32585,"depth":95,"text":34600},{"id":45416,"depth":88,"text":45416},"记一下学习 PPO中学习到的东西",{"date":45439,"image":45440,"alt":45441,"tags":45442,"published":91},"8 Mar 2026","/blogs-img/blog2.jpg","PPO学习笔记",[45443,12741],"Reinforcement learning","/blogs/ppo",{"title":12748,"description":45437},"blogs/2. PPO学习笔记","pCK9BRxyBBqY_YKiJBmT8uZH6lLudARB31wOAuwxMuM",{"id":45449,"title":45450,"body":45451,"description":45450,"extension":280,"meta":46396,"navigation":91,"ogImage":46398,"path":46402,"seo":46403,"stem":46404,"__hash__":46405},"content/blogs/3. 记录一下我的 zsh 配置.md","记录一下我的 zsh 配置",{"type":7,"value":45452,"toc":46373},[45453,45455,45458,45462,45465,45468,45471,45474,45477,45517,45524,45528,45531,45534,45581,45595,45612,45622,45625,45628,45631,45634,45640,45643,45663,45666,45669,45671,45690,45707,45709,45716,45721,45739,45742,45755,45778,45782,45785,45788,45793,45796,45800,45887,45891,46092,46096,46264,46267,46270,46277,46285,46288,46291,46294,46297,46300,46349,46355,46358,46365,46370],[10,45454,12],{"id":12},[14,45456,45457],{},"最近想写点东西记录一下自己折腾这些工具的经历，之前一直没想到具体写什么。今年实验室服务器迁移，刚好需要重新配一遍 zsh，干脆顺手把整套方案写下来，省得下次换机器再翻一堆旧博客。",[10,45459,45461],{"id":45460},"什么是-zsh","什么是 Zsh",[14,45463,45464],{},"Zsh 也是一种 shell，可以看成是一个更好用的 Bash——完全兼容 Bash 的语法，但补全、插件、主题方面强了不止一个量级。目前 macOS 的默认 shell 已经是 Zsh 了。配上插件之后，用起来在 Linux 下能省一大堆敲键盘的功夫。",[14,45466,45467],{},"Zsh 通过大部分主流 Linux 发行版的包管理器都能直接装。要是在没 sudo 权限的服务器上，也可以用 conda 装一份用户级别的，绕开权限问题。",[10,45469,45470],{"id":45470},"oh-my-zsh",[14,45472,45473],{},"Zsh 固然好用，但初始时只有白板没有任何插件，离理想的使用体验还差点意思，这里就轮到 oh-my-zsh 登场了。",[14,45475,45476],{},"oh-my-zsh 是社区为 zsh 收集的一个第三方插件和主题合集，国内可以通过南京大学镜像站快速安装：",[54,45478,45480],{"className":56,"code":45479,"language":58,"meta":59,"style":59},"git clone https://mirror.nju.edu.cn/git/ohmyzsh.git\ncd ohmyzsh/tools\nREMOTE=https://mirror.nju.edu.cn/git/ohmyzsh.git sh install.sh\n",[49,45481,45482,45493,45501],{"__ignoreMap":59},[63,45483,45484,45487,45490],{"class":65,"line":66},[63,45485,45486],{"class":69},"git",[63,45488,45489],{"class":77}," clone",[63,45491,45492],{"class":77}," https://mirror.nju.edu.cn/git/ohmyzsh.git\n",[63,45494,45495,45498],{"class":65,"line":88},[63,45496,45497],{"class":9544},"cd",[63,45499,45500],{"class":77}," ohmyzsh/tools\n",[63,45502,45503,45506,45508,45511,45514],{"class":65,"line":95},[63,45504,45505],{"class":200},"REMOTE",[63,45507,204],{"class":196},[63,45509,45510],{"class":77},"https://mirror.nju.edu.cn/git/ohmyzsh.git",[63,45512,45513],{"class":69}," sh",[63,45515,45516],{"class":77}," install.sh\n",[14,45518,45519,45520,45523],{},"通过这三条指令可以在境内快速装好 oh-my-zsh，安装过程中也会问你要不要把 zsh 设为默认 shell。要是当时手抖没设置，事后用 ",[49,45521,45522],{},"chsh -s $(which zsh)"," 也能补救。",[10,45525,45527],{"id":45526},"powerlevel10k","Powerlevel10k",[14,45529,45530],{},"oh-my-zsh 自带的主题我感觉还是不够美观、不够能折腾，于是这里再引入一个 Powerlevel10k 主题。在众多主题中，Powerlevel10k 因为美观和高度可定制脱颖而出，截至 2026 年 4 月 28 日，它在 GitHub 上已累计收获 54k 个 star。",[14,45532,45533],{},"国内安装可以用 gitee 的镜像：",[54,45535,45537],{"className":56,"code":45536,"language":58,"meta":59,"style":59},"git clone --depth=1 https://gitee.com/romkatv/powerlevel10k.git \"${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k\"\n",[49,45538,45539],{"__ignoreMap":59},[63,45540,45541,45543,45545,45548,45551,45553,45556,45559,45562,45565,45567,45569,45571,45573,45576,45579],{"class":65,"line":66},[63,45542,45486],{"class":69},[63,45544,45489],{"class":77},[63,45546,45547],{"class":73}," --depth=1",[63,45549,45550],{"class":77}," https://gitee.com/romkatv/powerlevel10k.git",[63,45552,11338],{"class":207},[63,45554,45555],{"class":77},"${",[63,45557,45558],{"class":200},"ZSH_CUSTOM",[63,45560,45561],{"class":196},":-",[63,45563,45564],{"class":200},"$HOME",[63,45566,2889],{"class":196},[63,45568,1142],{"class":77},[63,45570,45470],{"class":200},[63,45572,2889],{"class":196},[63,45574,45575],{"class":200},"custom",[63,45577,45578],{"class":77},"}/themes/powerlevel10k",[63,45580,214],{"class":207},[14,45582,45583,45584,45587,45588,45591,45592,12841],{},"之后修改 ",[49,45585,45586],{},"~/.zshrc","，把 ",[49,45589,45590],{},"ZSH_THEME"," 改成 ",[49,45593,45594],{},"powerlevel10k/powerlevel10k",[54,45596,45598],{"className":56,"code":45597,"language":58,"meta":59,"style":59},"ZSH_THEME=\"powerlevel10k/powerlevel10k\"\n",[49,45599,45600],{"__ignoreMap":59},[63,45601,45602,45604,45606,45608,45610],{"class":65,"line":66},[63,45603,45590],{"class":200},[63,45605,204],{"class":196},[63,45607,208],{"class":207},[63,45609,45594],{"class":77},[63,45611,214],{"class":207},[14,45613,45614,45617,45618,45621],{},[49,45615,45616],{},"source ~/.zshrc"," 或者重启终端后，会自动进入 Powerlevel10k 的配置向导，跟着选自己喜欢的风格就行。后面想再调，随时 ",[49,45619,45620],{},"p10k configure"," 就能重来一遍。",[14,45623,45624],{},"为了能完整显示主题里的图标，建议装一款 Nerd Font 字体。我个人用的是 0xProto Nerd Font，大家也可以挑别人推荐的字体，比如 FiraCode、JetBrainsMono、MesloLGS 之类的都行。装完之后记得在终端模拟器（iTerm2 / Windows Terminal / Alacritty / Kitty 等）的设置里把字体切换过去。",[10,45626,45627],{"id":45627},"插件设置",[14,45629,45630],{},"主题搞定了，下一步就是装上几个真正能帮我们提高生产力的插件。",[30,45632,45633],{"id":45633},"zsh-autosuggestions",[14,45635,45636,45637,45639],{},"非常有用的一个插件，根据历史记录和补全信息在你输入时实时给出灰色提示，按 ",[49,45638,30740],{}," 就能接受补全，能省下不少重复敲键盘的功夫。",[14,45641,45642],{},"安装指令：",[54,45644,45646],{"className":56,"code":45645,"language":58,"meta":59,"style":59},"git clone https://github.com/zsh-users/zsh-autosuggestions $ZSH_CUSTOM/plugins/zsh-autosuggestions\n",[49,45647,45648],{"__ignoreMap":59},[63,45649,45650,45652,45654,45657,45660],{"class":65,"line":66},[63,45651,45486],{"class":69},[63,45653,45489],{"class":77},[63,45655,45656],{"class":77}," https://github.com/zsh-users/zsh-autosuggestions",[63,45658,45659],{"class":200}," $ZSH_CUSTOM",[63,45661,45662],{"class":77},"/plugins/zsh-autosuggestions\n",[30,45664,45665],{"id":45665},"zsh-syntax-highlighting",[14,45667,45668],{},"提供指令和文件路径的高亮显示，方便你在按下回车之前就检查指令和路径有没有写错，相当于给终端加了层\"实时校对\"。",[14,45670,45642],{},[54,45672,45674],{"className":56,"code":45673,"language":58,"meta":59,"style":59},"git clone https://github.com/zsh-users/zsh-syntax-highlighting $ZSH_CUSTOM/plugins/zsh-syntax-highlighting\n",[49,45675,45676],{"__ignoreMap":59},[63,45677,45678,45680,45682,45685,45687],{"class":65,"line":66},[63,45679,45486],{"class":69},[63,45681,45489],{"class":77},[63,45683,45684],{"class":77}," https://github.com/zsh-users/zsh-syntax-highlighting",[63,45686,45659],{"class":200},[63,45688,45689],{"class":77},"/plugins/zsh-syntax-highlighting\n",[45691,45692,45693],"blockquote",{},[14,45694,45695,45696,45698,45699,45702,45703,45706],{},"小坑提醒：",[49,45697,45665],{}," 必须放在 ",[49,45700,45701],{},"plugins=(...)"," 列表的 ",[11217,45704,45705],{},"最后一项","，否则可能会和其他插件打架。",[30,45708,45486],{"id":45486},[14,45710,45711,45712,45715],{},"oh-my-zsh 自带的插件，把大量常用 git 指令简化成了别名，具体怎么用可以参考 ",[49,45713,45714],{},"$ZSH/plugins/git/git.plugin.zsh"," 这个文件。",[14,45717,45718],{},[323,45719],{"alt":45486,"src":45720},"zsh/git.png",[14,45722,45723,45724,45727,45728,45731,45732,45735,45736,235],{},"如图所示，alias 表示别名，等号左边是缩写指令，右边是原指令，比如用 ",[49,45725,45726],{},"gst"," 就能代替 ",[49,45729,45730],{},"git status","，用 ",[49,45733,45734],{},"gco"," 代替 ",[49,45737,45738],{},"git checkout",[30,45740,45741],{"id":45741},"启用插件",[14,45743,45744,45745,45747,45748,45751,45752,45754],{},"最后别忘了在 ",[49,45746,45586],{}," 里把上面三个插件加到 ",[49,45749,45750],{},"plugins"," 列表里，然后 ",[49,45753,45616],{}," 让配置生效：",[54,45756,45758],{"className":56,"code":45757,"language":58,"meta":59,"style":59},"plugins=(git zsh-autosuggestions zsh-syntax-highlighting)\n",[49,45759,45760],{"__ignoreMap":59},[63,45761,45762,45764,45766,45768,45770,45773,45776],{"class":65,"line":66},[63,45763,45750],{"class":200},[63,45765,204],{"class":196},[63,45767,1078],{"class":2483},[63,45769,45486],{"class":77},[63,45771,45772],{"class":77}," zsh-autosuggestions",[63,45774,45775],{"class":77}," zsh-syntax-highlighting",[63,45777,2545],{"class":2483},[10,45779,45781],{"id":45780},"kitty","Kitty",[14,45783,45784],{},"kitty 是一个基于 OpenGL 的可编程终端模拟器，适用于 Linux 与 macOS，具有真彩色，连字支持，键盘输入和图像渲染的协议扩展。它还提供了类似于 GNU Screen 或 tmux 的平铺功能。",[14,45786,45787],{},"kitty 在 ~/.config/kitty/kitty.conf 中存储配置，可以调整字体，颜色，光标，透明度和回滚行为。",[14,45789,45790],{},[323,45791],{"alt":45780,"src":45792},"zsh/kitty.png",[14,45794,45795],{},"我的项目结构如图所示，接下来会放出我的配置文件：",[30,45797,45799],{"id":45798},"kittyconf","kitty.conf",[54,45801,45805],{"className":45802,"code":45803,"language":45804,"meta":59,"style":59},"language-conf shiki shiki-themes github-light dracula","include themes/matugen.conf\nwindow_padding_width 5\nhide_window_decorations no\nbackground_opacity 0.8\nfont_family JetBrains Maple Mono \nfont_size 13.5\nremember_window_size no\nconfirm_os_window_close 0\nshell /bin/zsh --login\ncursor_trail 1 \ncursor_shape block\nshell_integration no-cursor\n# BEGIN_KITTY_THEME\n# Matugen\ninclude current-theme.conf\n# END_KITTY_THEME\n","conf",[49,45806,45807,45812,45817,45822,45827,45832,45837,45842,45847,45852,45857,45862,45867,45872,45877,45882],{"__ignoreMap":59},[63,45808,45809],{"class":65,"line":66},[63,45810,45811],{},"include themes/matugen.conf\n",[63,45813,45814],{"class":65,"line":88},[63,45815,45816],{},"window_padding_width 5\n",[63,45818,45819],{"class":65,"line":95},[63,45820,45821],{},"hide_window_decorations no\n",[63,45823,45824],{"class":65,"line":108},[63,45825,45826],{},"background_opacity 0.8\n",[63,45828,45829],{"class":65,"line":113},[63,45830,45831],{},"font_family JetBrains Maple Mono \n",[63,45833,45834],{"class":65,"line":2548},[63,45835,45836],{},"font_size 13.5\n",[63,45838,45839],{"class":65,"line":2556},[63,45840,45841],{},"remember_window_size no\n",[63,45843,45844],{"class":65,"line":2567},[63,45845,45846],{},"confirm_os_window_close 0\n",[63,45848,45849],{"class":65,"line":2584},[63,45850,45851],{},"shell /bin/zsh --login\n",[63,45853,45854],{"class":65,"line":2600},[63,45855,45856],{},"cursor_trail 1 \n",[63,45858,45859],{"class":65,"line":2608},[63,45860,45861],{},"cursor_shape block\n",[63,45863,45864],{"class":65,"line":2625},[63,45865,45866],{},"shell_integration no-cursor\n",[63,45868,45869],{"class":65,"line":2631},[63,45870,45871],{},"# BEGIN_KITTY_THEME\n",[63,45873,45874],{"class":65,"line":2656},[63,45875,45876],{},"# Matugen\n",[63,45878,45879],{"class":65,"line":2669},[63,45880,45881],{},"include current-theme.conf\n",[63,45883,45884],{"class":65,"line":2907},[63,45885,45886],{},"# END_KITTY_THEME\n",[30,45888,45890],{"id":45889},"current-themeconf","current-theme.conf",[54,45892,45894],{"className":45802,"code":45893,"language":45804,"meta":59,"style":59},"cursor #e0e2e8\ncursor_text_color #c2c7ce\n\nforeground            #e0e2e8\nbackground            #101418\nselection_foreground  #23323f\nselection_background  #b8c8da\nurl_color             #99ccfa\n\n# black\ncolor8   #8c9198\ncolor0   #4c4c4c\n\n# red\ncolor1   #ac8a8c\ncolor9   #c49ea0\n\n# green\ncolor2   #8aac8b\ncolor10  #9ec49f\n\n# yellow\ncolor3   #aca98a\ncolor11  #c4c19e\n\n# blue\n/* color4  #8f8aac */\ncolor4  #99ccfa\ncolor12 #a39ec4\n\n# magenta\ncolor5   #ac8aac\ncolor13  #c49ec4\n\n# cyan\ncolor6   #8aacab\ncolor14  #9ec3c4\n\n# white\ncolor15   #e7e7e7\ncolor7  #f0f0f0\n",[49,45895,45896,45901,45906,45910,45915,45920,45925,45930,45935,45939,45944,45949,45954,45958,45963,45968,45973,45977,45982,45987,45992,45996,46001,46006,46011,46015,46020,46025,46030,46035,46039,46044,46049,46054,46058,46063,46068,46073,46077,46082,46087],{"__ignoreMap":59},[63,45897,45898],{"class":65,"line":66},[63,45899,45900],{},"cursor #e0e2e8\n",[63,45902,45903],{"class":65,"line":88},[63,45904,45905],{},"cursor_text_color #c2c7ce\n",[63,45907,45908],{"class":65,"line":95},[63,45909,92],{"emptyLinePlaceholder":91},[63,45911,45912],{"class":65,"line":108},[63,45913,45914],{},"foreground            #e0e2e8\n",[63,45916,45917],{"class":65,"line":113},[63,45918,45919],{},"background            #101418\n",[63,45921,45922],{"class":65,"line":2548},[63,45923,45924],{},"selection_foreground  #23323f\n",[63,45926,45927],{"class":65,"line":2556},[63,45928,45929],{},"selection_background  #b8c8da\n",[63,45931,45932],{"class":65,"line":2567},[63,45933,45934],{},"url_color             #99ccfa\n",[63,45936,45937],{"class":65,"line":2584},[63,45938,92],{"emptyLinePlaceholder":91},[63,45940,45941],{"class":65,"line":2600},[63,45942,45943],{},"# black\n",[63,45945,45946],{"class":65,"line":2608},[63,45947,45948],{},"color8   #8c9198\n",[63,45950,45951],{"class":65,"line":2625},[63,45952,45953],{},"color0   #4c4c4c\n",[63,45955,45956],{"class":65,"line":2631},[63,45957,92],{"emptyLinePlaceholder":91},[63,45959,45960],{"class":65,"line":2656},[63,45961,45962],{},"# red\n",[63,45964,45965],{"class":65,"line":2669},[63,45966,45967],{},"color1   #ac8a8c\n",[63,45969,45970],{"class":65,"line":2907},[63,45971,45972],{},"color9   #c49ea0\n",[63,45974,45975],{"class":65,"line":4848},[63,45976,92],{"emptyLinePlaceholder":91},[63,45978,45979],{"class":65,"line":5124},[63,45980,45981],{},"# green\n",[63,45983,45984],{"class":65,"line":5130},[63,45985,45986],{},"color2   #8aac8b\n",[63,45988,45989],{"class":65,"line":5136},[63,45990,45991],{},"color10  #9ec49f\n",[63,45993,45994],{"class":65,"line":5142},[63,45995,92],{"emptyLinePlaceholder":91},[63,45997,45998],{"class":65,"line":5163},[63,45999,46000],{},"# yellow\n",[63,46002,46003],{"class":65,"line":5182},[63,46004,46005],{},"color3   #aca98a\n",[63,46007,46008],{"class":65,"line":5201},[63,46009,46010],{},"color11  #c4c19e\n",[63,46012,46013],{"class":65,"line":5206},[63,46014,92],{"emptyLinePlaceholder":91},[63,46016,46017],{"class":65,"line":5222},[63,46018,46019],{},"# blue\n",[63,46021,46022],{"class":65,"line":5228},[63,46023,46024],{},"/* color4  #8f8aac */\n",[63,46026,46027],{"class":65,"line":5234},[63,46028,46029],{},"color4  #99ccfa\n",[63,46031,46032],{"class":65,"line":5244},[63,46033,46034],{},"color12 #a39ec4\n",[63,46036,46037],{"class":65,"line":5268},[63,46038,92],{"emptyLinePlaceholder":91},[63,46040,46041],{"class":65,"line":5273},[63,46042,46043],{},"# magenta\n",[63,46045,46046],{"class":65,"line":5279},[63,46047,46048],{},"color5   #ac8aac\n",[63,46050,46051],{"class":65,"line":5293},[63,46052,46053],{},"color13  #c49ec4\n",[63,46055,46056],{"class":65,"line":5298},[63,46057,92],{"emptyLinePlaceholder":91},[63,46059,46060],{"class":65,"line":5304},[63,46061,46062],{},"# cyan\n",[63,46064,46065],{"class":65,"line":5319},[63,46066,46067],{},"color6   #8aacab\n",[63,46069,46070],{"class":65,"line":9624},[63,46071,46072],{},"color14  #9ec3c4\n",[63,46074,46075],{"class":65,"line":9630},[63,46076,92],{"emptyLinePlaceholder":91},[63,46078,46079],{"class":65,"line":9636},[63,46080,46081],{},"# white\n",[63,46083,46084],{"class":65,"line":9663},[63,46085,46086],{},"color15   #e7e7e7\n",[63,46088,46089],{"class":65,"line":9691},[63,46090,46091],{},"color7  #f0f0f0\n",[30,46093,46095],{"id":46094},"themesmatugenconf","themes/matugen.conf",[54,46097,46098],{"className":45802,"code":45893,"language":45804,"meta":59,"style":59},[49,46099,46100,46104,46108,46112,46116,46120,46124,46128,46132,46136,46140,46144,46148,46152,46156,46160,46164,46168,46172,46176,46180,46184,46188,46192,46196,46200,46204,46208,46212,46216,46220,46224,46228,46232,46236,46240,46244,46248,46252,46256,46260],{"__ignoreMap":59},[63,46101,46102],{"class":65,"line":66},[63,46103,45900],{},[63,46105,46106],{"class":65,"line":88},[63,46107,45905],{},[63,46109,46110],{"class":65,"line":95},[63,46111,92],{"emptyLinePlaceholder":91},[63,46113,46114],{"class":65,"line":108},[63,46115,45914],{},[63,46117,46118],{"class":65,"line":113},[63,46119,45919],{},[63,46121,46122],{"class":65,"line":2548},[63,46123,45924],{},[63,46125,46126],{"class":65,"line":2556},[63,46127,45929],{},[63,46129,46130],{"class":65,"line":2567},[63,46131,45934],{},[63,46133,46134],{"class":65,"line":2584},[63,46135,92],{"emptyLinePlaceholder":91},[63,46137,46138],{"class":65,"line":2600},[63,46139,45943],{},[63,46141,46142],{"class":65,"line":2608},[63,46143,45948],{},[63,46145,46146],{"class":65,"line":2625},[63,46147,45953],{},[63,46149,46150],{"class":65,"line":2631},[63,46151,92],{"emptyLinePlaceholder":91},[63,46153,46154],{"class":65,"line":2656},[63,46155,45962],{},[63,46157,46158],{"class":65,"line":2669},[63,46159,45967],{},[63,46161,46162],{"class":65,"line":2907},[63,46163,45972],{},[63,46165,46166],{"class":65,"line":4848},[63,46167,92],{"emptyLinePlaceholder":91},[63,46169,46170],{"class":65,"line":5124},[63,46171,45981],{},[63,46173,46174],{"class":65,"line":5130},[63,46175,45986],{},[63,46177,46178],{"class":65,"line":5136},[63,46179,45991],{},[63,46181,46182],{"class":65,"line":5142},[63,46183,92],{"emptyLinePlaceholder":91},[63,46185,46186],{"class":65,"line":5163},[63,46187,46000],{},[63,46189,46190],{"class":65,"line":5182},[63,46191,46005],{},[63,46193,46194],{"class":65,"line":5201},[63,46195,46010],{},[63,46197,46198],{"class":65,"line":5206},[63,46199,92],{"emptyLinePlaceholder":91},[63,46201,46202],{"class":65,"line":5222},[63,46203,46019],{},[63,46205,46206],{"class":65,"line":5228},[63,46207,46024],{},[63,46209,46210],{"class":65,"line":5234},[63,46211,46029],{},[63,46213,46214],{"class":65,"line":5244},[63,46215,46034],{},[63,46217,46218],{"class":65,"line":5268},[63,46219,92],{"emptyLinePlaceholder":91},[63,46221,46222],{"class":65,"line":5273},[63,46223,46043],{},[63,46225,46226],{"class":65,"line":5279},[63,46227,46048],{},[63,46229,46230],{"class":65,"line":5293},[63,46231,46053],{},[63,46233,46234],{"class":65,"line":5298},[63,46235,92],{"emptyLinePlaceholder":91},[63,46237,46238],{"class":65,"line":5304},[63,46239,46062],{},[63,46241,46242],{"class":65,"line":5319},[63,46243,46067],{},[63,46245,46246],{"class":65,"line":9624},[63,46247,46072],{},[63,46249,46250],{"class":65,"line":9630},[63,46251,92],{"emptyLinePlaceholder":91},[63,46253,46254],{"class":65,"line":9636},[63,46255,46081],{},[63,46257,46258],{"class":65,"line":9663},[63,46259,46086],{},[63,46261,46262],{"class":65,"line":9691},[63,46263,46091],{},[10,46265,46266],{"id":46266},"一些软件",[30,46268,46269],{"id":46269},"nvm",[14,46271,46272,46273,46276],{},"非常好用的 node.js 版本管理器，无需 sudo 权限，能解决系统级 node 下 ",[49,46274,46275],{},"npm install -g"," 安装时各种权限问题。",[14,46278,46279,46280,46284],{},"参考 ",[38,46281,46282],{"href":46282,"rel":46283},"https://github.com/nvm-sh/nvm",[42]," 安装。",[30,46286,46287],{"id":46287},"conda",[14,46289,46290],{},"无需多言。",[30,46292,46293],{"id":46293},"fastfetch",[14,46295,46296],{},"获取本机发行版和硬件信息，比较酷炫，装逼使用。",[14,46298,46299],{},"可以进行美化，参考指令：",[54,46301,46303],{"className":56,"code":46302,"language":58,"meta":59,"style":59},"cd $HOME/.config\n\n# 下载并解压然后删除\nwget https://github.com/sqlsec/fastfetch/releases/download/v0.1/fastfetch-for-Linux.zip\nunzip fastfetch-for-Linux.zip && rm fastfetch-for-Linux.zip\n",[49,46304,46305,46315,46319,46324,46332],{"__ignoreMap":59},[63,46306,46307,46309,46312],{"class":65,"line":66},[63,46308,45497],{"class":9544},[63,46310,46311],{"class":200}," $HOME",[63,46313,46314],{"class":77},"/.config\n",[63,46316,46317],{"class":65,"line":88},[63,46318,92],{"emptyLinePlaceholder":91},[63,46320,46321],{"class":65,"line":95},[63,46322,46323],{"class":2507},"# 下载并解压然后删除\n",[63,46325,46326,46329],{"class":65,"line":108},[63,46327,46328],{"class":69},"wget",[63,46330,46331],{"class":77}," https://github.com/sqlsec/fastfetch/releases/download/v0.1/fastfetch-for-Linux.zip\n",[63,46333,46334,46337,46340,46343,46346],{"class":65,"line":113},[63,46335,46336],{"class":69},"unzip",[63,46338,46339],{"class":77}," fastfetch-for-Linux.zip",[63,46341,46342],{"class":2483}," && ",[63,46344,46345],{"class":69},"rm",[63,46347,46348],{"class":77}," fastfetch-for-Linux.zip\n",[14,46350,46351,46352],{},"效果如图：\n",[323,46353],{"alt":46293,"src":46354},"zsh/fastfetch.png",[30,46356,46357],{"id":46357},"onefetch",[14,46359,46360,46361,46364],{},"获取当前 git 项目信息，使用时需保证当前目录已经 git 初始化（即包含 ",[49,46362,46363],{},".git"," 目录）。",[14,46366,46351,46367],{},[323,46368],{"alt":46357,"src":46369},"zsh/onefetch.png",[266,46371,46372],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sCLZk, html code.shiki .sCLZk{--shiki-default:#6F42C1;--shiki-dark:#50FA7B}html pre.shiki code .sEzAm, html code.shiki .sEzAm{--shiki-default:#032F62;--shiki-dark:#F1FA8C}html pre.shiki code .sDgm9, html code.shiki .sDgm9{--shiki-default:#005CC5;--shiki-dark:#8BE9FD}html pre.shiki code .scEb-, html code.shiki .scEb-{--shiki-default:#24292E;--shiki-dark:#BD93F9}html pre.shiki code .szJfE, html code.shiki .szJfE{--shiki-default:#D73A49;--shiki-dark:#FF79C6}html pre.shiki code .soDru, html code.shiki .soDru{--shiki-default:#005CC5;--shiki-dark:#BD93F9}html pre.shiki code .sMWOi, html code.shiki .sMWOi{--shiki-default:#032F62;--shiki-dark:#E9F284}html pre.shiki code .scbbO, html code.shiki .scbbO{--shiki-default:#24292E;--shiki-dark:#F8F8F2}html pre.shiki code .sfgPZ, html code.shiki .sfgPZ{--shiki-default:#6A737D;--shiki-dark:#6272A4}",{"title":59,"searchDepth":88,"depth":88,"links":46374},[46375,46376,46377,46378,46379,46385,46390],{"id":12,"depth":88,"text":12},{"id":45460,"depth":88,"text":45461},{"id":45470,"depth":88,"text":45470},{"id":45526,"depth":88,"text":45527},{"id":45627,"depth":88,"text":45627,"children":46380},[46381,46382,46383,46384],{"id":45633,"depth":95,"text":45633},{"id":45665,"depth":95,"text":45665},{"id":45486,"depth":95,"text":45486},{"id":45741,"depth":95,"text":45741},{"id":45780,"depth":88,"text":45781,"children":46386},[46387,46388,46389],{"id":45798,"depth":95,"text":45799},{"id":45889,"depth":95,"text":45890},{"id":46094,"depth":95,"text":46095},{"id":46266,"depth":88,"text":46266,"children":46391},[46392,46393,46394,46395],{"id":46269,"depth":95,"text":46269},{"id":46287,"depth":95,"text":46287},{"id":46293,"depth":95,"text":46293},{"id":46357,"depth":95,"text":46357},{"date":46397,"image":46398,"alt":46399,"tags":46400,"published":91},"28 April 2026","/blogs-img/blog3.jpg","Zsh 配置",[285,46401],"折腾笔记","/blogs/zsh",{"title":45450,"description":45450},"blogs/3. 记录一下我的 zsh 配置","HJxt7N1Hp2UuZpzLJWJCQuLpJNFDWThzx-XI5ReXWIw",{"id":46407,"title":46408,"body":46409,"description":46570,"extension":280,"meta":46571,"navigation":91,"ogImage":283,"path":46577,"seo":46578,"stem":46579,"__hash__":46580},"content/blogs/4. Agent 相关的折腾记录（持续更新）.md","Agent 相关的折腾记录（持续更新）",{"type":7,"value":46410,"toc":46557},[46411,46413,46416,46419,46422,46426,46429,46433,46436,46439,46443,46446,46450,46453,46456,46459,46463,46466,46469,46476,46479,46490,46494,46497,46500,46503,46506,46510,46521,46543],[10,46412,12],{"id":12},[14,46414,46415],{},"最近终于静下心来认真研究 Agent 生态。在此之前一直没下场深入折腾，主要是觉得这个领域生态变化太快、技术栈也不够成熟，迭代节奏完全跟不上。直到最近，Agent 相关的概念（Tools、MCP、Skills、Subagents、Hooks 等）逐渐收敛到一个相对稳定的形态，于是开始全面尝试。",[10,46417,46418],{"id":46418},"介绍一下我有哪些工具",[14,46420,46421],{},"截至目前，我体验过 Claude Code、Codex、Cursor，最近还试用了 SOLO Trae（支持国产）。下面分别说一下感受。",[30,46423,46425],{"id":46424},"claude-code","Claude Code",[14,46427,46428],{},"目前我用过编程体验最好的一个 Agent。聪明程度在一众 Coding Agent 里算是天花板级别，插件生态成熟，可以与 Cursor / VSCode 等编辑器协同使用：在终端里启动 Claude Code 后，能直接选中编辑器里的某段代码作为上下文同步过去；它给出的修改建议也能在编辑器里直观地 review 与 apply，整体闭环非常顺滑。",[30,46430,46432],{"id":46431},"codex","Codex",[14,46434,46435],{},"量大管饱，模型水平也相当能打，开 high reasoning 写代码完全顶得住。相比之下生态略弱一些，更偏\"原生 IDE 助手\"路线。",[14,46437,46438],{},"桌面端不错，浏览器自动化操作效果还挺好。",[30,46440,46442],{"id":46441},"cursor","Cursor",[14,46444,46445],{},"中规中矩。用 GPT codex 系列经常出现\"降智\"，Claude Sonnet 与 Opus 又容易过度思考；简单任务用 auto 模式既快又便宜。Tab 补全确实惊艳，对于\"手写古人\"来说值得一试。",[30,46447,46449],{"id":46448},"solo-trae","SOLO Trae",[14,46451,46452],{},"目前模型免费，可以用来跑一些轻量任务。问题在于自定义模型能力较弱、调用 Skill 还需要显式指定，加之经常需要排队，整体体验仍有较大提升空间。",[14,46454,46455],{},"最近 SOLO 更新了，可以使用制定模型了，热门国产模型基本都部署了，现在还是免费用，感觉平时也不会触发排队，比之前好多了，支持国产。",[14,46457,46458],{},"5 月底尝试了一下操作浏览器功能，非常拉，而且只有 auto 能操作浏览器，目前来看拉完了，远不如 codex。",[10,46460,46462],{"id":46461},"skills","Skills",[14,46464,46465],{},"最近发现 Skills 能显著扩展模型的能力边界——它本质上是一份\"在特定场景下注入到上下文的提示词包\"，让 Agent 在面对垂直任务时能够直接调用既定的方法论与工具链。后续打算系统挖掘一下 Skills 的设计模式与最佳实践。",[46467,46468],"hr",{},[14,46470,46471,46472,46475],{},"现在的 Skills 基本都能在 ",[38,46473,46474],{"href":46474},"skills.sh"," 这个网站上下载并获取，下面提到的 skills 也都是在这上面获取的。",[14,46477,46478],{},"发现一些比较好用的 Skills，下面简单介绍一下：",[12764,46480,46481,46484,46487],{},[12767,46482,46483],{},"Superpowers：耳熟能详了，性能非常强。",[12767,46485,46486],{},"find-skills：自动查找相关的 Skills。",[12767,46488,46489],{},"prompt-engineering-patterns：提示词设计范式。",[10,46491,46493],{"id":46492},"mcp","MCP",[14,46495,46496],{},"MCP（Model Context Protocol）方面体验还不算多。目前高频使用的是 Notion 的 MCP，可以让 AI 直接把整理好的内容写入 Notion 工作区，大大减少了\"复制 → 粘贴 → 排版\"的人肉环节。",[14,46498,46499],{},"最近使用了 CodeGraph，也是以 MCP 的形式提供服务，网上很流行，刚开始体验，使用一段时间再来谈谈自己的感受。",[10,46501,46502],{"id":46502},"杂谈",[14,46504,46505],{},"我还使用 CC-Switch 来管理多个 API 渠道。Claude Code 的主力渠道是公益站 AnyRouter，唯一的痛点是稳定性偏弱；不过最近恢复正常了，准备再战一波。",[10,46507,46509],{"id":46508},"zcf","ZCF",[14,46511,46512,46513,46516,46517,46520],{},"ZCF，全称 ",[11217,46514,46515],{},"Zero-Config Code Flow","，是一个面向专业开发者的 CLI 工具，目标是在几分钟内完成 Claude Code 与 Codex 的端到端环境初始化。通过 ",[49,46518,46519],{},"npx zcf"," 即可一站式完成：配置目录创建、API/代理接入、MCP 服务接入、工作流导入、输出风格与记忆配置，以及常用工具安装。",[14,46522,46523,46524,46527,46528,46531,46532,46535,46536,46539,46540,46542],{},"最近体验了一下 ",[49,46525,46526],{},"zcf:init-project","，发现确实比 Claude Code 与 Codex 自带的 ",[49,46529,46530],{},"init"," 强不少。它会生成总体级与目录级的 ",[49,46533,46534],{},"CLAUDE.md"," 作为指导文件，还会生成 ",[49,46537,46538],{},".claude/index.json"," 作为文件索引并记录修改时间；之后每次用 Claude Code 修改代码，都会顺带同步更新 ",[49,46541,46534],{},"，把\"何时改了什么\"完整记录下来。这种做法极大提升了 AI 代码的可维护性，避免出现\"AI 拉了一大坨，谁也不敢动\"的局面。代价是会增加 token 消耗，但在我看来收益远大于成本。",[14,46544,46545,46546,46549,46550,4629,46553,46556],{},"zcf 在安装时还可以可选地安装一些 MCP 服务。我个人常用的组合是 ",[11217,46547,46548],{},"spec-workflow","、",[11217,46551,46552],{},"context7",[11217,46554,46555],{},"open-websearch"," 这三个，不过最近 spec-workflow 经常连不上，后面要找时间排查一下。",{"title":59,"searchDepth":88,"depth":88,"links":46558},[46559,46560,46566,46567,46568,46569],{"id":12,"depth":88,"text":12},{"id":46418,"depth":88,"text":46418,"children":46561},[46562,46563,46564,46565],{"id":46424,"depth":95,"text":46425},{"id":46431,"depth":95,"text":46432},{"id":46441,"depth":95,"text":46442},{"id":46448,"depth":95,"text":46449},{"id":46461,"depth":88,"text":46462},{"id":46492,"depth":88,"text":46493},{"id":46502,"depth":88,"text":46502},{"id":46508,"depth":88,"text":46509},"记录自己折腾一些智能体相关技术的经历",{"date":46572,"image":283,"alt":46573,"tags":46574,"published":91},"30 April 2026","Agent",[46573,46401,46575,46493,46462,46576],"workflow","harness","/blogs/agent",{"title":46408,"description":46570},"blogs/4. Agent 相关的折腾记录（持续更新）","GM8kVLKc843iTi26s3XbF7eoeI8M8LbH9gEsGsp5qBU",{"id":46582,"title":46583,"body":46584,"description":46637,"extension":280,"meta":46638,"navigation":91,"ogImage":283,"path":46642,"seo":46643,"stem":46644,"__hash__":46645},"content/blogs/5. OpenMP 学习笔记.md","OpenMP 学习笔记",{"type":7,"value":46585,"toc":46632},[46586,46588,46591,46595,46600,46609,46611],[10,46587,12],{"id":12},[14,46589,46590],{},"这学期选修了并行计算这门课程，但学期过半一点还没开始学（），之前的实验啥的全仰仗 GPT 大人，最近开始打算好好学一学，先从课程的第一部分：CPU 并行学习。",[10,46592,46594],{"id":46593},"openmp","OpenMP",[45691,46596,46597],{},[14,46598,46599],{},"OpenMP（Open Multi-Processing）是一套支持跨平台共享内存方式的多线程并发的编程API，使用C,C++和Fortran语言，可以在大多数的处理器体系和操作系统中运行，包括Solaris, AIX, HP-UX, GNU/Linux, Mac OS X, 和Microsoft Windows。包括一套编译器指令、库和一些能够影响运行行为的环境变量。",[14,46601,46602,46603,46608],{},"这个介绍摘自 ",[38,46604,46607],{"href":46605,"rel":46606},"https://zh.wikipedia.org/wiki/OpenMP",[42],"维基百科 OpenMP","，在目前的并行计算框架中，OpenMP 是最容易上手的一个框架，通过许多高层抽象隐藏了实现的具体细节，只需要串行程序中加入OpenMP预处理指令, 就可以实现串行程序的并行化。",[10,46610,45416],{"id":45416},[12764,46612,46613,46619,46625],{},[12767,46614,46615],{},[38,46616,46583],{"href":46617,"rel":46618},"https://parallel.zhangjikai.com/openmp.html",[42],[12767,46620,46621],{},[38,46622,46624],{"href":46605,"rel":46623},[42],"维基百科",[12767,46626,46627],{},[38,46628,46631],{"href":46629,"rel":46630},"https://www.bilibili.com/video/BV1SW411s7ST",[42],"Introduction to OpenMP - Tim Mattson (Intel)",{"title":59,"searchDepth":88,"depth":88,"links":46633},[46634,46635,46636],{"id":12,"depth":88,"text":12},{"id":46593,"depth":88,"text":46594},{"id":45416,"depth":88,"text":45416},"2026年春季学期并行计算课程的OpenMP学习笔记。",{"date":282,"image":283,"alt":46639,"tags":46640,"published":91},"并行计算学习笔记",[285,46641],"并行计算","/blogs/openmp",{"title":46583,"description":46637},"blogs/5. OpenMP 学习笔记","3cqC4qDQLnD-zXRvzydKPsXhvOzyB39QnVWQ326QAKQ",{"id":4,"title":5,"body":46647,"description":279,"extension":280,"meta":46823,"navigation":91,"ogImage":283,"path":288,"seo":46825,"stem":290,"__hash__":291},{"type":7,"value":46648,"toc":46813},[46649,46651,46653,46655,46657,46659,46661,46663,46668,46672,46714,46716,46718,46722,46724,46735,46737,46739,46751,46753,46755,46787,46791,46797,46799,46811],[10,46650,12],{"id":12},[14,46652,16],{},[14,46654,19],{},[10,46656,22],{"id":22},[14,46658,25],{},[14,46660,28],{},[30,46662,33],{"id":32},[14,46664,36,46665,44],{},[38,46666,43],{"href":40,"rel":46667},[42],[14,46669,47,46670,52],{},[49,46671,51],{},[54,46673,46674],{"className":56,"code":57,"language":58,"meta":59,"style":59},[49,46675,46676,46688,46692,46702,46706],{"__ignoreMap":59},[63,46677,46678,46680,46682,46684,46686],{"class":65,"line":66},[63,46679,70],{"class":69},[63,46681,74],{"class":73},[63,46683,78],{"class":77},[63,46685,82],{"class":81},[63,46687,85],{"class":77},[63,46689,46690],{"class":65,"line":88},[63,46691,92],{"emptyLinePlaceholder":91},[63,46693,46694,46696,46698,46700],{"class":65,"line":95},[63,46695,98],{"class":69},[63,46697,78],{"class":77},[63,46699,82],{"class":81},[63,46701,105],{"class":77},[63,46703,46704],{"class":65,"line":108},[63,46705,92],{"emptyLinePlaceholder":91},[63,46707,46708,46710,46712],{"class":65,"line":113},[63,46709,116],{"class":69},[63,46711,119],{"class":77},[63,46713,105],{"class":77},[14,46715,124],{},[30,46717,127],{"id":127},[14,46719,130,46720,134],{},[49,46721,133],{},[30,46723,138],{"id":137},[14,46725,141,46726,147,46729,153,46732,159],{},[38,46727,146],{"href":144,"rel":46728},[42],[38,46730,152],{"href":150,"rel":46731},[42],[38,46733,158],{"href":156,"rel":46734},[42],[10,46736,162],{"id":162},[14,46738,165],{},[54,46740,46741],{"className":56,"code":168,"language":58,"meta":59,"style":59},[49,46742,46743],{"__ignoreMap":59},[63,46744,46745,46747,46749],{"class":65,"line":66},[63,46746,175],{"class":69},[63,46748,74],{"class":73},[63,46750,180],{"class":77},[10,46752,183],{"id":183},[14,46754,186],{},[54,46756,46757],{"className":56,"code":189,"language":58,"meta":59,"style":59},[49,46758,46759,46773],{"__ignoreMap":59},[63,46760,46761,46763,46765,46767,46769,46771],{"class":65,"line":66},[63,46762,197],{"class":196},[63,46764,201],{"class":200},[63,46766,204],{"class":196},[63,46768,208],{"class":207},[63,46770,211],{"class":77},[63,46772,214],{"class":207},[63,46774,46775,46777,46779,46781,46783,46785],{"class":65,"line":88},[63,46776,197],{"class":196},[63,46778,221],{"class":200},[63,46780,204],{"class":196},[63,46782,208],{"class":207},[63,46784,211],{"class":77},[63,46786,214],{"class":207},[14,46788,232,46789,235],{},[49,46790,211],{},[14,46792,238,46793,241,46795,245],{},[49,46794,133],{},[49,46796,244],{},[14,46798,248],{},[54,46800,46801],{"className":56,"code":251,"language":58,"meta":59,"style":59},[49,46802,46803],{"__ignoreMap":59},[63,46804,46805,46807,46809],{"class":65,"line":66},[63,46806,258],{"class":69},[63,46808,261],{"class":73},[63,46810,264],{"class":77},[266,46812,268],{},{"title":59,"searchDepth":88,"depth":88,"links":46814},[46815,46816,46821,46822],{"id":12,"depth":88,"text":12},{"id":22,"depth":88,"text":22,"children":46817},[46818,46819,46820],{"id":32,"depth":95,"text":33},{"id":127,"depth":95,"text":127},{"id":137,"depth":95,"text":138},{"id":162,"depth":88,"text":162},{"id":183,"depth":88,"text":183},{"date":282,"image":283,"alt":5,"tags":46824,"published":91},[285,286,287],{"title":5,"description":279},1780561597817]